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PREFACE 

This  document  is  the  final  technical  report  (CDRL  A003)  for  the  Factors  in 
Software  Quality  Study,  contract  number  F030602-76-C-0417.  The  contract  was 
performed  in  support  of  the  U.S.  Air  Force  Electronic  Systems  Division's 
(ESO)  and  Rome  Air  Development  Center's  (RADC)  mission  to  provide  standards 
and  technical  guidance  to  software  acquisition  managers. 

The  report  consists  of  three  volumes,  as  follows: 

Volume  I Concept  and  Definitions  of  Software  Quality 

Volume  II  Metric  Data  Collection  and  Validation 

Volume  III  Preliminary  Handbook  on  Software  Quality  for  an 
Acquisition  Manager 

\ 

The  objective  of  the  study  was  to  establish  a concept  of  software  quality 
and  provide  an  Air  Force  acquisition  manager  with  a mechanism  to  quantita- 
tively specify  and  measure  the  desired  level  of  quality  in  a software  product. 
Software  metrics  provide  the  mechanism  for  the  quantitative  specification  and 
measurement  of  quality. 

This  seccid  volume  describes  the  application  of  the  metrics  to  software 
products  and  the  validation  of  the  metrics'  relationship  to  software 
quality. 
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SECTION  7 

RELATIONSHIP  OF  METRICS  TO  QUALITY  FACTORS 
7.1  CONCEPT  OF  RELATIONSHIP 

The  hierarchical  framework  which  has  evolved  supports  the  simple,  understand- 
able, and  logical  relationships  of  the  components  of  the  software  quality 
concept.  It  also  supports  the  mathematical  formulation  of  the  relationship 
between  the  metrics  and  the  quality  factors. 

A software  development  can  be  envisioned  as  a process  which  is  controlled  by 
management  (both  contractor  management  and  AF  SPO  personnel).  This  control 
is  exercised  through  reviews,  status  reporting,  and  software  products  (Section 
5,  Interim  Report  #2)  delivered  during  the  development  effort  (Figure  7.1-1). 
Currently,  the  major  emphasis  of  the  control  is  to  evaluate  the  schedule  and 
cost  performance  and  to  determine  the  functional  correctness  of  the  software 
being  developed.  The  concept  underlying  software  quality  metrics  is  to  use 
these  control  vehicles  to  provide  an  indication  (and  therefore  a mechanism 
of  control)  of  the  quality  of  the  software  product  to  be  delivered. 

These  software  qualities  or  characteristics  which  go  beyond  the  technical 
mission  - qualities  such  as  reliability,  maintainability,  usability,  test- 
ability, and  portability,  which  have  been  defined  in  previous  sections  - 
have  been  recognized  in  recent  years  as  a necessary  concern  for  software 
development  managers. 

This  recognition  has  come  about  because  of  many  instances  in  which  not  con- 
sidering factors  such  as  these  has  driven  total  project  costs  well  beyond 
initial  estimates.  It  has  been  found  that  the  costs  throughout  the  total 
life  cycle  are  more  affected  by  the  characteristics  of  the  software  system 
than  by  the  mission-oriented  functions  performed  by  the  software  system. 

Large  software  systems  have  sometimes  proven  untestable,  unmodifiable,  and 
largely  unusable  by  operations  personnel  because  of  the  characteristics  of 
the  software. 


Software  Oevelofment  Process  Control 


The  metrics  that  have  been  established  provide  quantitative  measures  of 
specific  software  attributes.  At  any  specific  time  during  the  software 
development,  a set  of  metrics  can  be  applied  to  available  review  material, 
documents,  and  code.  Table  6.2-1  identifies  which  metrics  were  applied 
during  the  thrde  phases  of  requirements  analysis,  design,  and  progranmi ng . 

When  the  metrics  are  applied,  the  resulting  measurements  can  be  viewed  as 
an  n-tuple: 

(m"! , m^ , • • • , 

Each  element,  m. , of  this  n-tuple  represents  a quantitative  measure  of  the 
system  with  respect  to  a specific  metric  or  software  attribute. 

Certain  subsets  of  this  n-tuple  relate  to  specific  software  quality  factors. 

For  example,  metrics  i = 1 to  k may  relate  to  maintainability.  The  subset 
of  the  n-tuple 

(m^ , m2,  ...» 

then,  are  the  metrics  which  Table  6.3-1  identifies  as  related  to  maintain- 
ability. This  relationship  can  be  viewed  as  a function  relating  the 
measurement- tuple  to  a rating  of  the  specific  quality  factor: 

fCm^ , m2,  . . . , ^ 

where  rj^  is  a rating  of  the  maintainability  of  the  software. 

The  definitions  of  the  quality  factors  support  the  concept  of  a rating,  e.g., 
the  rating  for  the  quality  factor,  maintainability,  would  be  in  terms  of  the 
amount  of  effort  (man-days)  required  to  maintain  the  software. 

A preliminary  identification  of  the  nature  of  the  relationship  f(  ) was  the 
goal  of  the  fourth  phase  of  this  contract  effort.  The  relationship  is 
called  the  normalization  function. 

The  derivation  of  mathematically  complete,  generally  applicable  functions  were 
beyond  the  scope  of  this  effort.  The  limiting  factor  was  the  size  and  nature  of 
our  sample.  This  aspect  of  the  study  is  discussed  in  paragraph  7.2.  The  procedure 


for  the  derivation  of  the  normalization  functions  (paragraph  7.3).  the  concept 
of  their  use  through  a figure  of  merit  procedure  (paragraph  7.5).  and  the 
methodology  of  validating  these  relationships  (paragraph  7.4)  are  discussed  In 
this  report  and  specific  examples  from  our  experience  are  presented. 

The  metric  n-tuple  has  considerable  value  from  a quality  assurance  viewpoint. 

An  analogy  can  be  drawn  with  the  set  of  Indicator  lights  In  a cockpit  of  an 
airplane.  If  a particular  Indicator  light  flashes,  this  Imnedlately  Identifies 
a specific  characteristic  which  Is  beyond  acceptable  limits  or  has  reached 
a level  at  which  attention  should  be  focused  on  that  characteristic.  There 
may  be  a sound  reason  for  the  Indicator  to  be  flashing,  not  necessarily  result- 
ing from  an  underlying  problem,  but  a justification  should  be  established. 

This  particular  use  of  the  metric  n-tuple.  without  the  precise  relationship 
provided  by  a normalization  function.  Is  explored  and  discussed  further  In  the 
conclusion  of  this  report  and  In  Volume  III. 

7.2  DATA  USED  FOR  THIS  STUDY 

Two  large-scale  software  syst«n  developments  for  the  Air  Force  were  used  as 
the  data  bases  for  application  of  the  metrics.  The  two  systems  were  chosen 
because  they  represent  large-scale  software  developments,  were  developed 
according  to  military  standards,  and  represent  two  different  applications. 

Figure  7.2-1  presents  some  of  the  general  characteristics  of  the  two  systems. 

System  A Is  a command  and  control  system  developed  In  JOVIAL  (J4).  Periodically, 
a new  version  of  the  system  Is  developed  In  response  to  changes  In  hardware 
and  software  requirements.  The  data  base  used  represents  a recent  formal 
product  delivery  of  the  system.  The  system  represents  an  application  with 
which  both  the  customer  and  GE  have  had  considerable  experience.  The  system 
has  an  excellent  operational  history. 

System  B Is  a data  base  management  system  developed  In  JOVIAL  (J4).  The  data 
base  used  Is  the  Initial  software  development  of  the  system.  It  typified  the 
development  of  a new  capability  and  exhibited  several  significant  problems 
during  the  design  and  development  phases.  The  system  provides  the  capabilities 
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Figure  7.2-1  Air  Force's  Data  Bases  Chosen 


to  update,  delete,  and  modify  portions  of  a large  data  base  as  well  as 
selectively  list,  retrieve,  or  compare  two  data  bases. 


r 


•3 

The  data  base  for  each  system  consists  of  the  following: 

§ Documents  - The  complete  set  of  documents  identified  in 
Appendix  B,  including  the  requirements  specification,  design 
specificationv,  test  plans,  user  manual,  interface  control 
document,  etc.  were  available  for  analysis. 

• Review  Material  - The  documentation  prepared  for  and  the  recorded 
proceedings  of  the  various  reviews  identifed  in  Appendix  B were 
part  of  the  data  base. 

• Source  Code  - The  source  code  listings  of  each  program  in  both 
systems  were  available  both  in  hardcopy  and  on  magnetic  tape. 

• Problem  Reports  and  Configuration  Management  Reports  - The 
complete  set  of  problem  reports  identified  in  Appendix  B were 
available  as  well  as  the  configuration  management  report  which 
maintains  a log  and  status  of  the  problem  reports.  Figure  7.2-2 
presents  examples  of  each  of  these  reports. 

The  Design  Problem  Report,  DPR  (item  G),  identifies  problems  the 
progranmer  and  customer  have  encountered  with  the  system  analyst's 
design.  The  Software  Problem  Report,  SPR  (item  C),  identifies  soft- 
ware problems  encountered  with  a program.  The  Software  Analysis 
Report,  SAR  (item  D),  provides  the  scope  of  the  problem  and  recommends 
the  solution/action.  The  Modification  Transmittal  Memorandum,  MTM 
(item  E),  COMPOOL  Change  Request,  CCR  (item  F),  and  the  Data  Base 
Change  Request,  (DBCR,  not  shown),  indicate  solutions/changes  imple- 
mented. The  Configuration  Management  (CM)  Status  Update  Listing  (item  A) 
provides  time-to-fix  statistics  by  tracking  the  maintenance  efforts.  The 
Summary  Listing  (item  B)  will  be  discussed  in  more  detail  in  Section  8 
since  it  provided  automated  metric  data. 
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Figure  7.2-2  Problem  Reports  and  Configuration  Management 


Figure  7.2-3  DPR/SPR/MTM  History  For  Selected  Data  Bases 


For  the  selected  sample  of  modules,  the  corresponding  problem  reports 
were  extracted  and  categorized.  The  categorization  was  accomplished  by 
analyzing  the  problem  and  solution  described  on  the  problem  reports  and 
grouping  the  problem  according  to  Table  7.2-1.  Also,  data  on  man-power 
expenditure  to  fix  problems,  to  make  changes,  etc.  was  extracted  from 
the  configuration  management  system  and  categorized  according  to 
Table  7.2-1.  It  was  felt  that  man-power  expenditures  represented  the 
severity  or  cost  of  problems  more  than  the  number  of  problems. 

• Design  Charts  - As  part  of  the  design  docunents,  overview  and  detailed 
design  charts  are  provided.  For  System  B,  these  design  charts  were 
available  In  machine  readable  form,  which  facilitated  automated 
metric  analysis. 

e Metric  Information  - Considerable  metric  data  was  available  as  part 
of  the  data  bases.  The  sumnary  listings  (Item  B In  Figure  7.2-2) 
provided  a statistical  profile  of  each  program  for  the  two  systems. 

Such  statistics  as  the  number  of  cards,  statements,  procedures,  de- 
claratives, comments,  IFs,  FORs,  direct  code  statements  (assmably 


Table  7.2-1  Problem  Report  and  Man-Power  Expenditure  Categorization 


CATEeORY  BY 
QUALITY  FACTOR 

EXPLANATION 

t CORRECTNESS 

The  function  which  the  software  Is  to  perform  Is 
Incorrect.  The  rating  Is  In  terms  of  effort  required 
to  fix. 

• RELIABILITY 

The  software  does  not  function  as  expected.  The 
rating  Is  in  terms  of  effort  required  to  fix. 

• EFFICIENCY 

The  software  does  not  meet  performance  (speed,  stor- 
age) requirements.  The  rating  Is  In  terms  of  effort 
required  to  fix. 

• INTEGRITY 

The  software  does  not  provide  required  security. 

The  rating  Is  In  terms  of  effort  required  to  fix. 

• USABILITY 

There  Is  a problem  related  to  operation  of  the  soft- 
ware, the  user  Interface,  or  the  Input/output.  The 
rating  Is  in  terms  of  effort  required  to  fix. 

e MAINTAINABILITY 

The  rating  Is  In  terms  of  effort  required  to  correct 
any  of  the  above  problems. 

e FLEXIBILITY 

The  rating  Is  In  terms  of  effort  required  to  make  a 
modification  due  to  a change  In  specifications. 

• TESTABILITY 

The  rating  is  in  terms  of  effort  required  to  test 
changes  or  fixes. 

e REUSABILITY 

The  rating  Is  in  terms  of  effort  required  to  use 
software  in  a different  application. 

• PORTABILITY 

The  rating  is  In  terms  of  effort  required  to  convert 
the  software  to  operate  In  a different  environment. 

• INTEROPERABILITY 

The  rating  Is  In  terms  of  effort  required  to  couple 
the  system  to  another  system. 
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language),  GOTOs,  breaks  from  loops,  operands,  operators,  delimiters, 
etc.  were  available  for  the  two  systems.  This  metric  data  was  oriented 
primarily  to  the  source  code.  This  metric  data,  as  well  as  all  of  the 
other  metrics  established  during  this  effort,  are  covered  In  more 
detail  In  Section  8. 

Each  of  these  Items  was  available  for  analysis.  Essentially,  they  were 
utilized  as  sources  for  metric  data  or.  In  the  case  of  problem  reports  and 
the  configuration  management  report,  as  sources  of  the  error  and  maintenance 
history  of  the  software.  While  our  data  bases  represent  an  extremely  compre- 
hensive set  of  data  about  a software  system  development,  some  difficulties 
were  encountered.  Several  previous  or  on-going  efforts  ([THAYT76],  [NELSR75], 
[SH00M75],  [WILLN76])  sponsored  by  RADC  have  very  ably  discussed  the  problems 
of  data  collection.  Basically,  the  following  problems  arise  because  the  data 
Is  collected  after  the  fact: 

1.  Large  volume  of  data  which  must  be  manually  analyzed 

2.  Completeness  and  validity  of  data  with  respect  to  goal  of  analysis 
is  difficult  to  determine. 

3.  Impact  on  production  process  and  personnel  must  be  kept  at  a minimum. 

4.  Interpretation  of  data  decreases  In  accuracy  as  the  age  of  the  data 
increases. 

The  Impact  of  item  1,  with  the  resources  available  In  this  study,  was  to  reduce 
the  number  of  modules  that  were  analyzed.  System-level  metrics  were  applied 
to  both  systems  but  module-level  metrics  were  only  applied  to  approximately 
40%  of  the  modules  of  both  systems.  The  subset  of  modules  were  chosen  to 
be  representative  of  the  systems.  An  equal  distribution  of  modules  which 
were  small  ( < 400  statements),  medium  (400  < n <800  statements),  and  large 
(>  800  statements)  In  size  and  which  had  a small  number  of  SPRs  ( < 10), 
a medium  number  (10<  n £.25),  and  a large  number  (>  25)  written  against 
them  were  chosen. 
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The  Impact  of  Items  2»  3,  and  4 above  on  our  study  was  minimal  because  of 
the  large  amount  of  data  and  complete  documentation  that  Is  collected  and 
generated  during  a software  development  In  our  environment. 
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Several  other  restrictions  were  Imposed  on  the  study  because  of  data  avail- 
ability. During  the  derivation  of  the  quality  factors  and  quality  metrics, 
a complete  view  of  software  was  taken.  However,  the  operational  and  mainte- 
nance historical  data  necessary  to  validate  all  of  the  quality  factors  was 
not  available.  For  example,  some  of  the  metrics  are  system-level  metrics 
only,  I.e.,  they  are  measured  at  the  system  level.  Since  only  two  system 
developments  were  used,  development  of  a normalization  function  and  validation 
of  Its  accuracy  was  not  possible.  Also,  the  two  systems  have  not  experienced 
all  of  the  activities  required  to  accumulate  historical  data  to  validate 
metrics  relating  to  several  of  the  quality  factors.  For  example,  neither 
system  has  been  converted  to  operate  In  another  environment.  Therefore,  a 
normalization  function  relating  the  metrics  with  the  quality  factor  portability 
was  not  possible.  Metrics  relating  to  Interoperability,  portability, 
reusability,  testability.  Integrity,  and  efficiency  could  not  be  analyzed 
because  historical  data  was  not  available. 

All  of  the  metrics  were  applied  to  obtain  experience  with  their  data  collection. 
This  experience  Is  described  in  Section  8 and  Appendix  D.  Table  7.2-1  Identi- 
fies which  quality  factors  and  their  related  metrics  were  supported  by  the 
data  available.  The  plus  (+}  sign  indicates  that  an  analysis  was  possible 
because  data  was  available  and  the  metric  could  be  applied  at  the  module 
level.  A zero  (0)  Indicates  that  either  the  metric  was  a system  level  metric 
and  therefore  the  sample  was  too  small  and/or  historical  data  was  not  avail- 
able to  conduct  an  analysis.  The  code  column  relates  the  metrics  to  the 
software  quality  metrics  table  6.2-1. 


INTEROPOlWam 


Table  7.2-2  Data  Support  for  Normalization  Function  Pevelopment  and  Validation  (Cont.) 


qUM.lTY  FACTMtS 


USER  INPUT  INTERFACE  CM.1  S 

USER  OUTPUT  INTERFACE  CM. 2 S 

SOFTNMRE  SYSTEM 
INDEPENDENCE 

MACHINE  INDEPENDENCE 

COWUNICATIONS 
COMMONALITY 

DATA  COWVWALITY 
CONCISENESS 


DC.1  SYS 
C0.1  SYS 


1 

o 

Ik 

UJ 

USABILITY 

INTEGRITY 

NA 

PA 

NA 

FLEXIBILITY 

TESTABILITY 

REUSABILITY 

PORTABILITY 

INTEROPERABILITY 

PA 

NA 

NA 

NA  I 

NA 

- SYSTEM-LEVEL  METRIC 

- MODULE-LEVEL  METRIC 

- DATA  AVAILABLE  | 

- DATA  PARTIALLY  AVAILABLE | 

- DATA  NOT  AVAILABLE  ) 

- ANALYSIS  POSSIBLE 

- ANALYSIS  NOT  POSSIBLE 


HISTORICAL  DATA  AVAILABLE 
TO  DEVELOP  NORMALIZATION 
FUNCTION  FOR  QUALITY  FACTOR 


A more  subtle  yet  very  significant  Impact  on  our  study  was  the  fact  that  we 
applied  the  metrics  well  after  the  system  had  been  delivered  and  was  opera- 
tional. Many  of  the  problems  (Tow  metric  scores)  which  would  have  been 
realized  had  the  metrics  actually  been  applied  during  the  development  were 
not  evident.  For  example,  had  we  applied  the  set  of  metrics  related  to 
design  at  the  time  of  the  COR.  significantly  different  metric  scores  than  the 
ones  recorded  In  this  study  would  have  been  realized.  Over  time,  many  of  the 
problems  have  been  Identified,  analyzed  and  corrected.  This  bias  that  was 
Introduced  Is  very  significant,  especially  to  the  metrics  applied  during  the 
requirements  and  design  phases.  The  metric  scores  can  be  assumed  to  be 
significantly  Inflated. 

Thus,  the  most  effective  and  accurate  means  of  applying  the  metrics  and  also 
establishing  normalization  functions  would  be  In  an  on-line  mode,  that  Is, 
applying  the  metrics  during  a software  development  effort,  tracking  error 
history,  and  then  accumulating  operational  and  maintenance  historical  data 
to  establish  normalization  functions.  The  data  would  be  current  and  therefore 
more  accurate  and  easier  to  collect  and  would  reflect  the  status  of  the  soft- 
ware development  In  terms  of  the  software  quality  metrics  more  realistically. 
This  on-line  application  of  the  metrics  Is  described  further  In  Section  8 v/hen 
the  application  of  the  metrics  during  this  effort  Is  discussed. 

7.3  NORMALIZATION  FUNCTION  DEVELOPMENT 

In  this  section,  a description  of  the  methodology  of  deriving  a normalization 
function  will  be  described  and  then  examples  will  be  provided.  Complete 
results  from  this  effort  are  contained  in  Appendix  C. 

The  methodology  Is  as  follows: 


I 
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• The  metric  n-tuple  for  a particular  phase  of  development  Is  applied 
to  the  available  software  products  (review  material,  documents,  code). 
This  process  Is  done  Initially  at  a module-by-module  basis  and  then 
at  a system  level.  The  application  of  each  metric  Is  described  In 
Section  8.  The  results  of  this  step  are  n-tuples  of  measurements 
for  each  module  and  for  each  system. 
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t Subsets  of  the  measureoient  n-tuples  which  relate  to  specific  quality 
factors  are  segregated.  For  example,  the  k-tuple,  (m^,  m^.  . . .m|^), 
which  represents  the  measurements  for  the  k metrics  which  relate  to  the 
quality  factor,  maintainability,  are  extracted  for  each  module  and  each  j 
system. 

e Data  which  represent  the  quality  factor  performance  or  rating  of 
the  Individual  modules  and  systems  are  collected.  For  example,  the 
amount  of  effort  expended  to  correct  fixes  to  each  module  and  system 
Is  collected  to  represent  a rating  of  the  maintainability  of  that 
module  or  system. 

e Using  the  measurement  k-tuple  as  independent  variables  and  the  ratings 
of  the  individual  modules  or  systems  as  dependent  variables,  a regres- 
sion analysis  Is  performed  to  derive  the  normalization  function. 

Linear  regression  analysis  was  performed  in  this  study.  There  is 
some  indication  that  in  a few  selected  cases  a nonlinear  function 
might  be  more  appropriate.  This  exploration  should  be  considered  in 
future  efforts.  The  resulting  function,  in  the  linear  regression 
case,  takes  the  form: 

r^  = aQ  + a^m^  + a2m2  + . . . aj^mj^ 

where  r^  is  the  predicted  rating  of  quality  factor  f,  given  the 

measurement  k-tuple  (m-j,  m^,  m^,  ....  mj^)  and  the  a^.  are  the 

regression  coefficients  derived  from  the  regression  analysis. 

These  weights  assigned  the  individual  metrics  reflect  their  predic- 
tive value  with  respect  to  the  particular  quality  factor.  Several 
iterations  of  this  procedure  are  required  to  eliminate  the  metrics 
which  do  not  show  significant  correlation.  If  time  had  permitted, 
initially  a complete  factors  analysis  would  have  been  performed  to 
group  related  metrics  with  specific  quality  factors.  This  was  done 
intuitively  during  the  process  of  establishing  the  metrics. 


There  is  a serious  misinterpretation  which  can  be  made  at  this  point. 

The  utility  of  the  derived  normalization  function  Is  very  dependent  upon 
the  sample  used.  In  the  case  of  this  study,  the  two  systems  used. 


while  two  different  applications,  were  developed  in  the  same  environment, 
according  to  very  strict  standards  and  conventions,  using  the  same  language, 
machine,  operating  system,  development  tools,  etc.  For  this  reason  many 
metrics,  when  applied  to  all  of  the  modules,  showed  no  variation  in  measure- 
ments. A simple  example  is  the  metric  (SI. 2),  use  of  a structured  language 
or  structured  preprocessor.  JOVIAL  (J4)  wa's  considered  a structured  language, 
although  according  to  a very  strict  definition  it  is  not,  because; 

• Several  of  the  structured  programming  constructs  are  implemented 
within  the  language. 

f It  is  a block  oriented  language 

e Our  standards  and  conventions  restrict  the  use  of  constructs  which 
violate  some  of  the  structured  programning  philosophies. 

Every  module,  then,  received  a score,  or  measurement,  of  1 for  this  metric. 
Since  this  measurement  showed  no  variation,  the  regression  analysis  indicated 
there  was  no  correlation  between  this  metric  and  the  quality  factor  rating. 

If  this  result  is  interpreted  absolutely,  then  one  could  conclude  that  the 
use  of  a structured  language  or  structured  preprocessor  has  no  effect  or 
correlation  with  the  resulting  maintainability  of  the  system.  This  is 
obviously  an  incorrect  conclusion.  What  the  result  does  mean  is  that  for 
the  application  in  which  the  metrics  were  applied  and  the  regression  analysis 
was  performed,  the  variability  in  the  maintainability  of  the  modules  in  a 
system  or  between  systems  is  not  a function  of  the  use  of  a structured 
language.  The  reason  is  that  the  use  of  a structured  language  is  a standard 
to  which  there  is  strict  adherence. 

Our  expectation  is  that  if  these  metrics  and  methodology  were  applied  to  other 
system  developments  in  other  environments  and,  for  this  particular  example, 
in  environments  where  the  use  of  a structured  language  or  structured  pre- 
processor varied,  a significant  correlation  between  this  metric  and  the 
resulting  rating  of  maintainability  would  be  indicated. 

Thus,  the  use  and  Interpretation  of  the  normalization  function  is  critical 
to  its  effectiveness.  This  concept  is  considered  in  more  depth  in  Section  9. 
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To  illustrate  the  methodology  an  example  will  be  presented  in  this  section. 
Complete  data  from  the  regression  analyses  are  provided  in  Appendix  C. 


Using  the  quality  factors  which  were  supported  with  historical  data  and 
for  the  metrics  which  could  be  applied  at  the  module  level  (see  Table  7.2-2), 
several  analyses  were  performed.  These  analyses  included  analyzing  indi- 
vidual metrics  versus  the  rating  of  a factor  and  a multiple  regression 
analysis  of  the  k-tuple  of  metrics  relating  to  a factor.  In  certain  cases 
individual  elements  of  a metric  were  also  analyzed  on  a single  basis  with 
a quality  factor  when  high  correlation  was  expected.  The  methodology  to 
perform  any  one  of  these  analyses  was  the  same.  An  example  of  the  analysis 
of  one  metric,  SD.2,  effectiveness  of  conments  measure,  and  its  relation- 
ship to  the  quality  factor,  maintainability,  will  be  given  to  illustrate 
the  methodology. 

A sample  of  modules  from  System  B was  used  to  develop  the  normalization 
function  relating  metric,  SD.2,  individually  to  maintainability. 

Standard  linear  regression  techniques  ([THAYT76],  [FLEIT66],  [LAB0V66],  [COOLWBZj 
[POOLL77],  [PADED56],  [KUESJ73])  were  used.  Routines  to  perform  the  analysis 
developed  on  a PDP  11/40  and  Tektronix  4051  terminal.  Since  the  metric  and 
the  quality  factor  rating  were  normalized  positive  valurs,  all  data  points 
fall  within  the  positive  quadrant  of  a graph.  The  regression  line  was 
forced  through  the  origin  to  support  this  concept. 


Using  the  metric  table.  Table  6.2-1,  the  measures  associated  with  the 
effectiveness  of  comments  metric  were  collected  from  a sample  of  modules. 
Historical  operations  and  development  data  was  used  to  determine  the 
number  of  fixes  to  each  module  in  the  sample  and  the  number  of  man-days 
expended  to  accomplish  these  fixes.  A rating  of  maintainability  was 
then  calculated  for  each  module  by  the  following  formula: 


where: 

MO^  > Total  number  of  man-days  expended  on  fixes  to  module  1. 

n^  = Total  number  of  fixes  to  module  1. 

rl  = Normalized  rating  of  maintainability  for  module  1. 

The  rating  of  maintainability  then  Is  based  on  the  average  number  of  man- 
days  expended  to  make  a fix  to  the  software. 

For  the  sample  chosen,  the  distribution  of  occurrence  for  the  metric 
value  and  the  rating  are  shown  In  Figure  7.3-1  and  Figure  7.3-2  respectively. 

The  histogram  is  generated  by  accumulating  the  number  of  data  point  falling 
in  the  interval  k < x^  < k + 0.1,  where  k = 0,  0.1,  0.2,...,  0.9  and  dividing  by 
the  total  number  of  modules  in  the  sample  to  arrive  at  a frequency  of  occur- 
rence figure. 

The  Independent  variable  Is  the  metric  value  determined  for  each  module. 

The  dependent  variable  Is  the  rating  value  determined  for  each  module. 

The  resulting  regression  equation  is  the  normalization  function.  Its  form 
In  the  case  of  one  Independent  variable  is: 

■■m  ■ Vf 

where  r»,  Is  the  predicted  rating  of  maintainability,  a^  Is  the  predictor 

” e SO  2 

coefficient  for  the  metric  value,  m^,  in  this  case  metric  SD.2,  ’ . 

Figure  7.3-3  Illustrates  the  regression  line  determined  for  this  metric: 


The  dashed  lines  represent  a 90  percent  confidence  Interval  for  the  sample. 
The  standard  error  of  estimate  Is  0.15.  The  correlation  coefficient  for 
the  regie  ' on  line  Is  0.92.  This  represents  a significant  correlation 
between  >ne  .netrlc  and  the  rating  of  maintainability. 
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Figure  7.3-1  Frequency  Distribution  for  Metric  SD.2 
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This  InforMtlon  Is  displayed  In  Table  7.3-1.  This  table  Is  an  excerpt 
froM  Table  C-8,  where  the  results  of  the  analyses  of  all  of  the  metrics 
related  to  maintainability  are  described. 

This  example  will  be  continued  In  paragraphs  7.4  and  7.5  to  Illustrate 
the  methodology  of  validating  the  metrics  and  arriving  at  a figure  of 
merit.  The  analysis  of  data  points  which  fall  outside  the  90  percent 
confidence  Interval  will  be  discussed  In  paragraph  7.4.  The  histograms 
are  provided  to  allow  some  evaluation  of  the  sample.  In  this  example, 
all  metric  values  were  above  0.5.  While  we  would  have  liked  to  had  a better 
range  of  metric  values  for  the  sample,  other  modules  measured  In  System  B 
fall  within  this  same  range.  This  represents  an  environmental  limitation 
to  our  study  because  comments  are  strongly  emphasized  by  our  standards 
and  conventions. 

Appendix  C contains  the  remainder  of  the  results  of  the  normalization 
function  development.  Tables  such  as  Table  7.3-1  provide  summary  results 
of  the  normalization  functions  for  the  groups  of  metrics  related  to  a 
quality  factor,  for  each  Individual  metric  related  to  a quality  factor, 
and  In  some  Instances,  selected  metric  elements  related  to  a quality  factor. 
Remarks  highlight  specific  results  In  Appendix  C.  Where  regression  analysis 
was  not  performed,  the  summarized  metric  values  are  provided  for  Informa- 
tion and  there  Is  an  Indication  as  to  the  reason  why  no  regression  analysis 
was  performed. 


Table  7.3-1  Regression  Analysis  Sumnary 


r 


SYSTEM  B 

METRIC 

SD.2 

MAINTAINABILITY 

RATING 

INDIVIDUAL 

METRIC 

ANALYSIS 

AVERAGE 

.75 

.32 

RANGE 

.5-1. 

.07-. 77 

STD  DEV 

.13 

.15 

PREDICTOR 

COEFFICIENT 

.46 

STANDARD 

ERROR  OF 
ESTIMATE 

.15 

CORRELATION 

COEFFICIENT 

.92 

I : 
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7.4  VALIDATION  PROCESS 

The  process  of  validating  a normalization  function  will  be  described  and 
examples  presented.  Appendix  C contains  a summary  of  the  complete  set 
of  data  used  during  the  validation  phase  of  this  effort. 

The  methodology  used  for  validating  a normalization  function  Is  as  follows: 

• The  normalization  function  derived  In  paragraph  7.3  predicts  a rating 
for  any  measurement  k-tuple.  There  Is  a certain  variance  associated 
with  the  predicted  rating  and  actual  rating. 

• a 90%  confidence  Interval  Is  determined  based  on  the  normalization 
function,  the  variance,  and  the  sample.  Another  subset  of  modules 

Is  then  plotted  and  depending  on  their  compliance  with  the  confidence 
Interval,  the  normalization  function  Is  accepted  or  rejected. 

• In  the  case  where  a normalization  function  Is  rejected,  several 
actions  can  be  taken: 

- Conduct  a factor  analysis  to  determine  the  minimum  number  of 
dimensions  needed  to  describe  the  relevant  Information  con- 
tained In  the  original  measurements. 

- Reevaluate  the  metrics  and  their  units  to  ensure  they  do  not 
present  possible  ambiguities  In  their  measurement  of  the  cri- 
terion or  quality  factor. 

- Evaluate  the  correlation  coefficients  of  the  metrics.  While  a 
metric  may  logically  be  an  Important  characteristic  of  a software 
product.  It  may  not  correlate  well  and  therefore  is  not  a consis- 
tent predictor  of  the  final  quality  of  the  software. 

- Reestablish  a normalization  function  by  utilizing  regression 
analysis  techniques. 

- Evaluate  the  quality  factor  rating  (Its  representative  distribution). 
It  may  not  be  linear  Itself  and  therefore  cause  problems  In  estab- 
lishing a linear  relationship  with  the  associated  metrics.  Consider- 
ation of  nonlinear  regression  will  be  given  in  this  case. 
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In  the  example  presented  in  paragraph  7.3,  a linear  function  for  the  metric 
SD.2,  effectiveness  of  comments,  and  a 90%  confidence  interval  were  deter- 
mined using  a subset  of  modules  of  System  B.  The  corresponding  metric  data 
and  maintainability  rating  data  were  compiled  for  a subset  of  modules  of 
System  A.  This  data  is  summarized  in  Table  7.4-1. 

Table  7.4-1  Validation  Data  Sunmary  for  Maintainability 


SYSTEM  A 

METRIC  SD.2 

Individual 

Average 

.77 

metric 

Range 

.6-.8 

analysis 

Std  dev 

.018 

Average 

.306 

Rating 

Range 

.21-. 40 

Std  dev 

.068 

Normalization 

function 

accepted/ 

rejected 


Accepted 


Plotting  the  points  on  the  graph  presented  in  Figure  7.3-3  results  in 
Figure  7.4-1.  All  of  the  data  points  fall  within  the  90%  confidence  inter- 
val. Recalculation  of  the  regression  line  using  all  of  the  data  points  does 
not  change  any  of  the  values  substantially.  Thus,  this  normalization  function 
is  considered  validated. 


Note  that  there  was  one  point  in  the  initial  sample  which  fell  outside  the 
90%  confidence  interval.  An  evaluation  of  this  module  revealed  that  while 
several  fixes  had  been  made  to  the  module,  they  all  were  related  to  a basic 
problem.  Thus,  the  average  time  to  make  the  several  fixes  was  quite  low, 
resulting  in  the  unusually  high  rating.  Based  on  this  explanation,  we  felt 
it  was  not  justified  to  reject  the  predictor  coefficient  determined.  During 
the  validation  for  several  metrics,  data  points  fell  outside  of  the  90% 
confidence  interval.  Those  modules  were  evaluated  for  any  abnormalities 
that  would  justify  their  deviation  from  the  norm.  Where  justifications  were 
not  found,  the  normalization  function  was  rejected.  In  the  cases  where  time, 
resources,  and  data  permitted,  the  steps  described  in  the  beginning  of  this 
paragraph  were  taken. 
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Figure  7.4-1  validation  of  SO. 2 


The  statements  made  in  paragraph  7.2  qualifying  the  interpretation  of  the 
normalization  function  should  be  reemphasized  here.  The  sample,  both  in 
size  and  variation  in  nature,  limits  the  general  applicability  and  precision 
of  the  results.  The  methodology  established  and  the  results  achieved, 
nevertheless,  are  valuable. 

For  instance,  of  the  three  metrics  quantifying  self-descriptiveness  (SD.l, 

50. 2,  SO. 3),  only  SO. 2,  effectiveness  of  comments,  correlated  well  with 
maintainability.  SD.l,  a measure  of  the  quantity  of  conments,  varied  insig- 
nificantly in  our  sample.  The  mean  was  .25  (the  percent  of  comnents  per 
card  was  25%),  ranging  essentially  between  .20  and  .30.  This  lack  of  varia- 
tion in  the  measurements  meant  no  correlation  with  maintainability  could  be 
established.  The  reason  for  this  lack  of  variation  is  our  standards  and 
conventions  establish  very  strict  guidelines  on  what  situations  should  be 
comnented.  This  does  not  mean  that  SD.l  would  not  be  a valuable  metric  in 
another  environment  where  the  standards  are  not  as  strict.  It  does  mean  that 
in  our  environment,  where  the  percentage  of  comments/cards  in  programs  are 
fairly  standard,  no  variation  in  maintainability  is  attributable  to  that  metric. 

50. 3,  descriptiveness  of  implementation  language,  did  not  correlate  well  with 
maintainability  either.  Again,  the  main  reason  was  the  lack  of  variation  in 
the  measurements  caused  by  our  strict  standards  on  the  format  of  the  programs 
and  naming  conventions  for  variables  and  a specific  software  support  tool 
(GEOIT)  used  to  preprocess  all  source  code  and  indent,  block,  and  number  the 
code  in  a logical,  standard  manner.  Variation  in  this  metric  would  be  found 
between  system  developments  or  in  an  environment  where  strict  standards  or 
automated  tools  are  not  used. 

The  summarized  data  and  results  of  the  validation  process  for  these  three 
metrics,  as  well  as  all  of  the  other  metrics  for  which  normalization  functions 
were  derived,  are  in  Appendix  C. 


7.5  FIGURE  OF  MERIT  PROCEDURE 

The  intent  in  deriving  a figure  of  merit  is  to  provide  the  SPO  with  an  overall 
measure  foi-  each  quality  factor.  The  figure  of  merit  will  be  normalized 
according  to  the  standard  units  of  measurement  chosen  for  that  factor.  In 
keeping  with  our  example,  the  figure  of  merit  for  maintainability  will  be  in 
terms  of  the  average  time  to  fix. 

The  figure  of  merit  procedure  is  basically  the  use  of  the  normalization  function 
as  a predictor  of  the  level  of  quality  being  achieved  for  a quality  factor. 

Thus,  if  a particular  measurement  k-tuple  is  applied  at  a specific  time  during 
the  development  phase,  the  values  obtained  cah  be  inserted  in  the  normalization 
function  for  that  quality  factor  for  that  phase  and  a figure  of  merit  is  determined. 


this  figure  of  merit  can  then  be  evaluated  relative  to  the  level  of  quality 
specified  by  the  customer.  If  the  figure  of  merit  is  below  the  specified 
level,  evaluation  and/or  corrective  actions  should  be  initiated.  If  the 
figure  Of  merit  is  above  the  specified  quality  level,  then  some  degree  of 
confidence  that  the  development  effort  is  progressing  satisfactorily  with 
respect  to  the  required  software  qualities  is  derived. 


Figure  7.5-1  continues  the  example  used  previously.  The  normalization 
function  is: 

y-  • AK  mSD.2 

If  the  system  (or  module)  measurement  for  SD.2  was  found  to  be  .75  during  the 
implementation  phase  of  the  development,  a predicted  rating  of  .345  results.  1 
the  SPO  had  specified  that  the  required  maintainability  of  the  system  was  an 
average  time-to-fix  of  one  man-week  (1/5  man-days  = * -2,  identified  in 

Figure  7.5-1  by  point  A),  he  has  approximately  an  84*  level  of  confidence  that 
the  maintainability  of  his  system  will  be  better  than  he  specified.  This 
figure  is  arrived  at  using  the  predicted  rating,  the  specified  rating,  and 
the  standard  error  of  estimate  determined  during  the  regression  analysis. 
Figure  7.5-2  illustrates  the  derivation  of  the  confidence  level. 
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SD.2  EFFECTIVENESS  OF  COMMENTS  MEASURE 
Figure  7.5-1  Figure  of  Merit  Procedure 


MEAN  * .345  (PREDICTED  RATING} 

STANDARD  DEVIATION  - .15  (STANDARD  ERROR  OF  ESTIMATE) 
LEVEL  OF  CONFIDENCE  - Pr  {x  > .2}  * .84  (SHADED  AREA) 


Figure  7.5-2  Determination  of  Level  of  Confidence 


This  relationship  between  the  figure  of  merit  derived  from  the  metrics  and 
the  quality  level  specified  by  an  SPO  is  elaborated  upon  in  Appendix  E, 
which  presents  a preliminary  handbook  for  SPOs  on  the  specification  of 
software  qualities. 


SECTION  8 


METRIC  DATA  COLLECTION 

8.1  METRIC  APPLICATION 

While  the  historical  data  and  sample  size  prevented  the  establishment  of 
normalization  functions  for  all  of  the  quality  factors,  a significant  benefit 
of  the  study  was  the  experience  gained  in  applying  all  of  the  metrics  to  the 
systems.  Whether  regression  analysis  was  to  be  performed  or  not,  the  metric 
data  was  collected  for  all  of  the  metrics. 

It  was  found  that  applying  the  metrics  to  the  various  software  products 
provided  considerable  insight  into  the  design  and  implementation  of  the 
software.  This  reflects  the  n-tuple  (indicator  lights)  concept  described 
in  paragraph  7.1. 

The  purpose  of  this  section  is  to  identify  where  the  metric  data  was  found, 
how  it  was  collected  during  this  study,  and  what  automated  tools  are  available 
that  could  be  used  to  collect  the  data  in  future  applications. 

During  this  effort  most  metric  data  was  collected  manually.  Some  automated 
tools  were  available  and  used.  Other  tools  were  identified  that  would  provide 
metric  data.  Table  8.1-1  provides  sunmary  information  on  the  collection 
techniques  used. 

Table  8.1-1  Summary  of  Collection  Techniques 
REQUIREMENTS  DESIGN 

25  108 


25  100 

- (0%) 8 (7%) 

6 (24%)  30  (28%) 


DEVELOPMENT 

PHASES 

Number  of 
elements 

Collected 
during  this 
study: 

Manually 

Automatically 

Can  be 

automatically 

checked 
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Thus,  over  40%  of  the  metric  data  can  be  collected  via  automated  means.  As 
more  formal  languages  and  techniques  evolve  for  preparing  requirements 
specifications  and  design  specifications,  a larger  percentage  of  automated 
collection  would  be  expected.  The  remaining  metric  d.  ta  which  Is  manually 
collected.  In  general,  can  be  done  quite  routinely  by  trained  personnel. 

There  Is  the  typical  10%  of  the  data  which  requires  a significant  data  collec- 
tion effort.  As  experience  with  the  metrics  Is  gained,  a determination  of  the 
significance  of  those  metrics  and  whether  they  are  worth  the  data  collection 
effort  required  can  be  made. 

Table  8.1-2  Identifies  the  sources  of  the  metric  data  and  the  number  of 
elements  which  use  that  source  for  metric  data,  e.g.,  the  system  requirement 
specification  Is  utilized  as  a source  of  data  for  23  measurements  (elements). 
More  than  one  source  may  be  used  to  arrive  at  a measurement,  so  the  totals 
between  this  table  and  Table  8.1-1  do  not  directly  correspond. 


Table  8.1-2  Source  Frequency 


SOURCES 

NUMBER  OF  TIMES 
USED  AS  SOURCE 

Software  System  R .'qulrements  Specification 

23 

Standards  and  Conventions 

8 

Preliminary  Design  Specification 

13 

Preliminary  Design  Review  Material 

2 

Detailed  Design  Specifications 

98 

Critical  Design  Review  Material 

2 

Validation  and  Acceptance  Test  Specification 

5 

User's  Manual /Operator's  Manual 

25 

Interface  Control  Document 

4 

Data  Base  Management  Plan 

3 

Problem  Reports 

2 

Source  Code 

135 

Training  Material 

2 

Programmer's  Notebook 

2 
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Manually 

25 
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The  frequency  with  which  the  sources  are  used  indicates  their  importance 
to  the  resulting  software  end  product  and  the  quantif lability  of  their 
contents.  For  example,  the  source  code  is  most  frequently  used  and  the 
detailed  design  specification  is  second  most  frequently  used.  The  importance 
of  the  user's/operator's  manual  is  indicated  by  its  relatively  high  use.  The 
relatively  high  use  of  the  software  system  requirement  specification  and  the 
preliminary  design  specification  emphasizes  the  fact  that  some  measures  can 
ba  applied  very  early  in  the  development  phase. 

Appendix  D provides  a detailed  examination  of  each  measure,  identifying  where 
it  was  collected  for  this  study  and  what  type  tool  is  available  to  automate 
its  collection.  The  next  two  paragraphs  briefly  describe  the  automated  tools 
used  to  collect  the  metric  data  during  this  study  (paragraph  8.2)  and  those 
tools  applicable  to  this  task  (paragraph  8.3)  but  not  available  during  this 
study. 

In  Appendix  0,  examples  are  used  to  highlight  the  procedures  and  tools  used 
to  determine  the  measures.  To  illustrate  the  contents  of  this  appendix,  the 
following  example  is  provided: 

An  element  of  the  design  structure  metric  is  based  on  the  number  of 
modules  which  do  not  have  a single  entrance  and  a single  exit  (SI. 1(6)). 
During  the  design  phase  of  a software  development,  an  example  of  an 
automated  tool  which  provides  data  for  this  measure  is  the  Integrated 
Software  Development  System  (GE/ISDS),  described  in  paragraph  8.2. 

Using  design  charts  in  machine  readable  form,  GE/ISDS  performs  various 
analyses  on  the  design  of  individual  programs.  One  such  analysis 
identifies  routines  which  have  multiple  entrances  and  exits.  An  example 
of  a design  chart  and  the  resulting  automated  analysis  is  shown  in 
Figure  8.1-1  . 

Appendix  D contains  many  other  such  examples. 
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Figure  8.1-T  Automated  Standards  Checking  During  Design 


8.2  TOOLS  USED  FOR  DATA  EXTRACTION 

The  following  software  support  tools  were  used  during  this  study  to  auto- 
matically collect  metric  data.  Examples  of  outputs  from  the  tools  are  In 
Appendix  D corresponding  to  the  measures  which  they  provided. 

A brief  overview  oriented  toward  describing  the  capabilities  of  each  tool 
Is  provided.  The  Intent  In  describing  these  tools  and  providing  examples 
of  their  output  In  Appendix  D Is  to  emphasize  that  automated  metric  appli- 
cation 1s  possible  early  In  the  development  phase. 

8.2.1  GE/INTEGRATED  SOFTWARE  DEVELOPMENT  SYSTEM  (GE/ISDS) 

GE/ISDS  Is  an  Integrated  system  of  software  support  tools  based  on  a common 
data  base  of  software  development  Information.  Current  capabilities  emphasize 
analyses  utilizing  machine  readable  design  charts.  Some  of  the  automated 
analyses  Include  analysis  of  the  design  charts  for  compliance  with  standards, 
flow  path,  minimum  number  of  tests  required,  and  connectivity.  Prototype 
versions  of  several  other  tools  include  methods  for  using  structured  pro- 
gramming constructs  In  flowcharts.  Interactive  data  base  usage/structure 
definition,  a measure  of  program  complexity  based  on  control  structure  and 
variable  usage,  and  a formalized  test  procedure  language  for  thorough 
testing  of  program  segments  ([CHANP76],  [RICHP74],  [RICHP76]). 


8.2.2  CODE  AUDIT  ROUTINES  (GJSUMRY/ATP) 

These  routines  provide  a profile  of  software  characteristics  for  JOVIAL  (J4) 
code.  Included  in  the  profile  for  each  routine  are  counts  of  the  number  of 
cards,  statements,  procedures,  declarations,  connents,  IPs,  FORs,  direct  code 
statements,  GOTOs,  breaks  from  loops,  operators,  operands,  delimiters,  and 
other  specified  JOVIAL  constructs  {[ALGEC77]). 


8.2.3  CONFIGURATION  MANAGEMENT  SYSTEM 

As  an  aid  to  the  strict  configuration  control  of  the  development  of  software  and 
any  changes  made,  this  system  maintains  a current  status  of  any  problem  reports 
recorded  against  a routine.  The  system  provides  a log  of  any  changes  made  and 
actions  taken  with  regard  to  the  software  being  developed. 


8.2.4  REQUIREMENTS  TRACE  ROUTINE 

This  routine  maintains  a current  list  of  perfomance  requirements  Identified 
with  Itemized  software  systms  requirement  specifications. 

The  Impact  of  using  these  tools  during  this  study  quite  significant.  Far 
less  data  collection  would  have  been  possible  If  all  data  collection  had  been 
done  manually.  The  advantages  In  accuracy  and  manpower  savings  of  automated 
data  collection  stresses  Its  Importance  to  the  application  of  metrics. 

8.3  OTHER  TOOLS  APPLICABLE  TO  METRIC  DATA  COLLECTION 
Several  other  software  support  tools  have  been  Identified  which  appear  to  be 
applicable  to  metric  data  collection.  A brief  description  of  several  will  be 
provided  in  this  section.  The  Intent  of  th’s  section  Is  not  to  provide  a support 
software  tools  survey,  so  the  descriptions  will  be  generic  In  nature.  Examples 
of  available  tools  will  be  mentioned. 

8.3.1  REQUIREMENTS  SPECIFICATION  LANGUAGE/ ANALYZER 

The  underlying  concept  of  this  tool  Is  that  if  the  requirements  specification 
is  written  In  a formal  language,  some  form  of  analyses  can  be  made  on  the 
specification.  The  analyses  that  can  be  done  that  relate  to  our  metrics  fall 
within  the  completeness  and  consistency  areas.  Examples  of  this  tool  are 
PSL/PSA  [TIECD76]  and  RSL  [BELLT76]. 

8.3.2  PROGRAM  DESIGN  LANGUAGE/ANALYZER 

Consistent  with  the  concept  expressed  above.  If  the  design  specification  is 
written  In  a formal  language  (PDL),  some  analyses  can  be  automatically  performed. 
GE/ISDS  provides  these  type  of  capabilities  based  on  design  charts.  Planned 
enhancements  are  to  provide  the  same  analysis  capabilities  for  a PDL.  Some 
examples  of  work  in  this  area  are  the  PDL  [PR0G75]  concept  originated  by  IBM 
and  the  HOS  concept  [HAMIM76]. 

8.3.3  AUTOMATED  VERIFICATION  SYSTEM 


These  support  software  tools  Involve  instrumenting  source  code  to  measure  test 
effectiveness.  The  structure  of  the  code,  as  well  as  path  usage  and  time  data. 
Is  analyzed.  Some  assistance  In  generating  test  data  Is  provided.  Examples  of 


tools  in  this  group  include  FLOW  [RICHP76].  ANALYZER  [ NBS74  ].  NODAL  [N0DA75]. 
PET  [ PET72  ].  and  JAVS  ([BROON76],  [MILLE74]). 

6.3.4  TEST  PROCEDURE  LANGUAGE 

This  tool  is  currently  under  development  and  is  based  on  the  use  of  a test  pro- 
cedure language  (TPL)  to  formally  state  and  document  test  procedures  and  a 
VERIFIER  to  apply  the  test  procedures  to  the  target  modules  or  system.  The 
test  procedures  are  a deliverable  product  of  the  software  development  process 
and  are  used  for  both  initial  checkout  and  subsequent  regression  testing  of 
target  program  modifications  [PANZD76]. 

8.3.5  EXECUTION  ANALYZER 

Considerable  information  is  gained  by  executing  code  under  various  loading 
conditions.  A post  execution  routine  could  provide  automated  analysis  and 
reports  of  pertinent  metric  data  based  on  the  execution.  Such  infomation 
as  run  time,  core  usage,  module  link-time  and  OS  link-time  are  some  of  the 
examples  indicated  in  Appendix  D that  could  be  reported  via  an  automated 
tool . 

8.3.6  CONSISTENCY  CHECKER 

This  type  tool  provides  the  capability  to  identify  various  consistency  measures 
relating  to  data,  variable  usage  and  initialization  as  well  as  others.  Con- 
sistency checking  can  be  done  at  the  code  level  [RAMAC75]  or  at  a specification 
level,  such  as  extensions  to  PSL/PSA  would  provide. 

8.3.7  DATA  BASE  ANALYZER 

Analysis  can  be  performed  on  the  data  base  via* tools  such  as  a data  definition 
language  processor  or  a data  base  optimizer.  The  analyses  center  on  data 
usage,  data  structure,  and  data  redundancy. 

These  tools  represent  a sample  of  those  available.  The  major  concern  would  be 
of  effectively  using  a subset  of  these  tools  in  a software  development  environ- 
ment. An  integrated  concept  would  be  required. 
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APPENDIX  C 


RESULTS  OF  DEVELOPMENT  AND  VALIDATION 
OF  NORMALIZATION  FUNCTIONS 

This  appendix  is  organized  as  follows: 

For  each  quality  factor: 

• Analysis  was  not  performed  on  certain  metrics.  These  metrics  are 
identified  and  reasons  why  they  were  not  used  is  given  according 
to  the  codes  described  in  Table  C-1. 

• Regression  analysis  was  performed  on  certain  individual  metrics. 

A summary  of  the  metric  scores  and  the  quality  factor  ratings 

for  the  quality  factor  ratings  for  the  subset  of  modules  (System  B) 
used  is  given  as  well  as  the  results  of  the  analysis:  predictor 
coefficient,  standard  error  of  estimate,  and  correlation  coefficient. 
The  results  are  plotted  on  graphs. 

• Based  on  these  results,  a second  subset  of  modules  (from  System  A) 
were  plotted  on  the  same  graphs  as  a validation  of  the  normaliza- 
tion functions  developed  in  the  above  step.  A sunrary  of  the  rating 
and  metric  scores  for  this  second  subset  is  given  for  comparison. 
Acceptance  or  rejection  of  the  normalization  function  is  indicated. 

• Based  on  these  results,  certain  metrics  were  chosen  to  be  used  in 


Table  C-1  ' Reasons  for  No  Analysis  or  Correlation 


COK 

EXPLANATION 

R1 

NOT  SUFFICIENT  VARIATION  IN  DATA  BASE  - Very  strict 
standards  or  a restriction  of  the  development  envi- 
ronment may  cause  a very  limited  range  of  metric 
scores  which  would  limit  statistical  significance. 

R2 

NO  HISTORICAL  DATA  AVAILABLE  - It  may  be  impossible 
to  derive  a rating  of  a quality  factor  because  sup- 
porting data  was  not  collected  or  the  system  had 
not  experienced  the  activity  represented  by  the 
quality  factor.  For  example,  if  a system  had  not 
been  moved  from  one  environment  to  another,  there 
would  be  no  data  to  derive  a rating  of  portability. 

R3 

DATA  BASE  DOES  NOT  SUPPORT  METRIC  DATA  COLLECTION  - 
Data  may  not  be  available  in  the  data  base  which 
is  required  to  determine  a metric.  For  example, 
programner  notebooks  which  contained  module  level 
testing  information  were  not  available,  therefore 
many  of  the  instrumentation  measures  could  not  be 
applied.  There  was  no  source  available  for  those 
metrics. 

R4 

SYSTEM  LEVEL  METRIC  - Since  only  two  systems  were 
used,  a larger  sairple  is  required. 

C-2 


TOLERANCE  CONTROL  MEASURE  (DESIGN) 


IW»UT^TA  ERROR  TOLERANCE 


Figure  C-4  Sl.ln  (Design)  Normalization  Function 


TOLERANCE  CONTROL  MEASURE  (IMPLEMENTATION) 


Figure 'C-7  SI. Ip  (iMplementation)  Nomalizetlon  Function 


SI  .4  fCASURE  OMIIflS  SimiCITY  (mPLEMEMTATION} 
Figure  C-9  S1.4p  (Implementation)  normal Ization  function 
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USABILITY 


MINTAIflMILITY 


STRUCTURE  MEASURE  (DESIGN) 


COMPLEXITY  MEASURE  (DESIGN  AND  IMPLEMENTATION) 

[.3u  (Design  and  Implementation)  Normalization  Function 


STRUCTURE  MEASURE  (IMPLEMENTATION) 


MODULAR  IMPLEMENTATION  MEASURE  (IMPLEMENTAnON) 

-14  N0.2|,  (Implementation)  Normalization  Function 


DESCRIPTIVENESS  OF  IMPLEMENTATION  LANGUAGE  MEASURE  (IMPLEMENTATION) 


GE.2  IMPLEMENTATION  FOR  GENERALITY  (DESIGN) 


MO. 2 MODULAR  IMPLEMENTATION  MEASURE  (IMPLEMENTATION) 


DESCRIPTIVENESS  OF  IMPLEMENTATION  LANGUAGE  (IMPLEMENTATION) 


REUSABILITY 


REUSABILITY  (Continued) 


INTEROPERABILITY  (Continued) 


APPENDIX  D 
METRIC  APPLICATION 


Each  metric  will  be  discussed  in  this  appendix  in  relation  to  where  it  was 
applied  (its  source)  and  how  it  was  collected.  Selected  examples  will  be 
given. 

The  metrics  will  be  covered  in  the  same  order  as  presented  in  Table  6.2-1. 
The  legend  presented  in  Table  0.1-1  applies  to  the  discussion: 


Table  D.1-1  Metric  Source  and  Tool  Legend 


Sources 

Code 

Software  System  Requirements  Specification 

SRS 

Standards  and  Conventions 

SC 

Documentation  Plan 

DP 

Management  Plan 

MP 

Preliminary  Design  Specification 

PDS 

Preliminary  Design  Review  Material 

PDR 

Detailed  Design  Specifications 

DDS 

Critical  Design  Review  Material 

CDR 

Validation  and  Acceptance  Test  Specification 

VATS 

Users  Manual /Operators  Manual 

UOM 

Interface  Control  Document 

ICD 

Configuration  Management  Plan 

CMP 

Data  Base  Management  Plan 

DBMP 

Problem  Reports 

PR 

Source  Code 

CODE 

Programmers  Notebook 

PNK 

Training  Material 

TM 

1 

Table  D.  1-1  Metric  Source  and  Tool  Legend  (Cont. ) 


( 


Tools/Techniques  Used* 

Code 

Manual  Inspection  or  Review 

MAN 

Configuration  Management  System 

CMS 

Requirements  Trace  Routine 

RTR 

6E/ Integrated  Software  Development  System 

ISDS 

GJSUMRY/ATP  - Code  Auditor 

GJS 

* A brief  description  is  provided  In  paragraph  6.2 


Where  manual  techniques  were  used  but  automated  tools  have  been  Identified 
that  would  assist  In  or  are  capable  of  providing  the  metric  data,  a code 
from  Table  D.1-2  Is  provided  In  parentheses.  These  are  briefly  described 
and  examples  given  in  paragraph  8.3. 


Table  D.1-2  Other  Data  Collection  Tools 


Automated  Tools  Available 

Code 

Requirements  Specification  Language/Analyzer 

PSL 

Automated  Verification  System 

AVS 

Test  Procedure  Language 

TPL 

Program  Design  Language/ Analyzer 

PDL 

Code  Auditor 

CA 

Execution  Analyzer 

EA 

Consistency  Checker 

CC 

Data  Definition  Language  Processor 

DDLP 

Data  Base  Optimizer 

DBO 

fH 
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Nrraic 

NEC 

INTS 

SOURCE 

TOOL 

SOURCE 

TOOL 

SOURCE 

TOOL 

CP.  1 COMPLETENESS  CHECKLIST: 

(1)  UiMinblauous  references  (Input,  func..on, 
output) . 

SRS 

NAN 

(PSD 

DOS 

NAN 

(POL) 

CODE 

NAN 

(CA) 

(2)  All  date  references  defined,  computed,  or 
obtained  from  an  external  source. 

SRS 

NAN 

(WL) 

DOS 

HAN 

(POL) 

CODE 

NAN 

(CA) 

(3)  All  defined  functions  used. 

SRS 

NAN 

(PSD 

DOS 

MAN 

(POL) 

CODE 

NAN 

(CA) 

(4)  All  referenced  functions  defined. 

SRS 

NAN 

(PSD 

DOS 

NAN 

(POL) 

CODE 

NAN 

(CA) 

(5)  All  conditions  and  processing  defined  ior 
each  decision  point. 

SRS 

NAN 

(PSD 

DOS 

ISOS 

CODE 

NAN 

(CA) 

(6)  All  defined  and  referenced  calling  sequence 
parameters  agree. 

DOS 

NAN 

CODE 

1 

NAN 

(7)  All  problem  reports  resolved. 

PR 

(CMS) 

PR 

(CHS) 

PR 

CNS 

(8)  Design  agrees  with  requirements. 

DOS 

SRS 

MAN 

irol:! 

(9)  Code  agrees  with  design. 

DOS 

CODE 

1 

(ISOS) 

(CA) 

Most  of  the  completeness  measures  (CP.1)  were  1,  I.e.,  no  deficiencies  were 
; 1 detected.  This  was  a function  of  applying  the  metrics  after  delivery.  Most 

I of  these  measures  were  applied  manually  during  this  effort.  The  use  of  a 

i formal  requirements  specification  language  and  formal  design  language  would 

' I greatly  enhance  the  automation  of  the  metric  data  collection  for  this  metric. 

Automation  Is  definitely  required  because  manual  determination  of  these 
! measures  during  an  on-going  project  would  be  a difficult  effort.  An  example 

of  an  automated  tool  used  during  this  effort  Is  the  GE/Integrated  Software 
I Development  System  (GE/ISOS)  (a  description  Is  In  paragraph  8.2).  GE/ISDS 

^ performs  analyses  on  design  charts  which  are  prepared  via  an  Interactive 

[ ; graphics  system  and  maintained  In  machine  readable  form.  An  example  output 

from  an  analysis  of  decision  points  (CP.l  (5))  Is  shown  In  Figure  D.2-1. 

s ♦ j 

: H A tool  such  as  PSL/PSA  (CARA)  provides  the  capability  to  determine  such 

* ' I 

I I measqrts  as  CP.l  (3)  and  CP.l  (4).  Figure  D.2-2  provides  an  excerpt  from 

r 'I  a configuration  management  report  which  was  used  to  determine  CP.l  (7). 
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Figure  D.2-1  Automated  Analysis  of  Decision  Points 
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Figure  D.2-2  Status  of  Problem  Reports 
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METRIC 


CS.  1 PROCEDURE  COMSISTENCY  MEASURE 

1(1)  Standard  design  representation 
/,  I modules  vloiate  ru1e\ 


(2)  Calling  sequence  conventions 
/,  I modules  violote  ru1e\ 


A I module* 
('■  total 

I Calling  sequ< 

* module! 

I tota 

I Input/output 

# nndule! 

' tot? 

Error  handlli 

(,  < module; 
' tala’ 


(3)  Input/output  conventions 
/i  * modules  violate  ru 
( total  il  modules 


tions 
te  ru1e\ 
ules  j 


(4)  Error  handling  conventions 
/,  < modules  violate  rule! 
( * total  # iDOdutes  : 


DESIGN 

SOURCE 

TOOL 

DOS 

ISOS 

DOS 

ISOS 

ICO 

DOS 

ISDS 

DOS 

NAN 

(ISDS) 

Enhancements  to  many  of  the  typical  currently  utilized  code  audit  routines 
could  display  for  easier  inspection  or  enforce  conventions  relating  to 
the  CS.l  elements  identified.  During  design,  GE/ISDS  is  an  example  of 
automated  analysis  performed  on  design  charts  relating  to  these  con- 
ventions. Enhancements  to  a tool  such  as  ISDS  could  provide  complete 
coverage  of  this  metric.  Examples  are  shown  in  Figure  D.3-1. 
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Figure  D.3-1  Automated  Consistency  Checks  During  Design 
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HCTAIC 

REgms 

OESISN 

SOURCE 

TOOL 

SOURCE 

Tool 

SOURCE 

TOOL 

CS.  2 DATA  CONSISTENCY  HEASURE 

(1)  SUndard  data  usage  representation 
/.  1 nodules  violate  rule\ 

(' total  fl'midul'e's — j 

ops 

hAa 

(isps) 

(2)  Naning  conventions 

/.  f nodules  violate  ru1e\ 

(’ total  ir'ibdulM — ) 

DOS 

NAN 

(ISOS) 

(OOLP) 

CODE 

NAN 

(DDLP) 

tCA) 

\ / 

(3)  Unit  consistency 

/.  f nodules  violate  ru1e\ 

^touvrasTos — ) 

DOS 

NAN 

CODE 

NAN 

(4)  Consistent  global  definitions 
/.  1 nodules  violate  rule  \ 

(’*  total  1 SoiuTel — j 

DOS 

time 

NAN 

CODE 

NAN 

(DDLP) 

(CA) 

\ w 

(S)  Data  type  consistency 

(■-  < ■*isia  im?'"’) 

DOS 

NAN 

CODE 

NAN 

(CA) 

I The  measures  dealing  with  data  consistency  (CS.2)  pose  a very  difficult 

I manual  data  collection  task.  The  use  of  an  automated  tool  Is  necessary  both 

\ from  a cost  to  collect  end  en  accuracy  standpoint.  The  use  of  a formal 

Program  Design  Language  would  enhance  the  ability  to  automate  the  collection 
of  this  data.  For  this  effort,  the  Data  Base  Management  Plan,  the  section 
of  the  Detailed  Design  Specifications  which  Identified  the  Internal 
variables  to  be  used  In  a routine,  and  the  organization  and  substantial 
comnentlng  of  the  code  aided  the  manual  collection  of  this  metric  data. 

A current  enhancement  to  the  design  charts  produced  during  contractual 
i efforts  to  Include  data  representations  will  allow  automated  checking  of 

j CS. 2(1),  as  shown  In  Figure  D.4-1. 


0.4  (continued) 
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Figure  0.4-1  Data  Representation  on  Flowcharts 

The  type  of  problem  that  should  be  identified  by  CS.2(4)»  consistent  global 
definitions,  is  illustrated  by  this  example; 

Routine  A 

C0^f10N/VAR/SUM,  OEV,  ROOT 
C0MM0N/MATRIX/X(15).  Y(25) 

Routine  B. 

COMMON/VAR/TOTAL,  ROOT,  OEV 
C0MM0N/MATRIX/X{25),  Z(15) 

The  Data  Base  Management  Plan  and  adherence  to  the  plan  in  the  design 
documents  is  the  key  to  prevention  of  these  types  of  errors. 
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METRIC 


IRC.  1 ACCURACY  CHECKLIST: 

(1)  Error  «ne1ys1s  perfonred  and  budgeted  to 
module. 

(2)  A definitive  statement  of  regulrenent  for 
accuracy  of  Inputs,  outputs,  processing, 
and  constants. 

(3)  Sufficiency  of  math  library. 

(4)  Sufficiency  of  numerical  methods. 

(5)  Execution  outputs  within  tolerances. 


REQKTS 


SOURCE  TOOL 


SRS 

SRS 


MAN 

MAN 


0ESI6N 


SOURCE  TOOL 


DOS 


DOS 


MAN 

(EA) 

NAN 


IMPLEMENTATION 


SOURCE  TOOL 


COOE 

CODE 


MAN 

NAN 

(EA) 


The  accuracy  checklist  measures  (AC.l)  require  a high  level  analyst  familiar 
with  the  mathematical  requirements  of  the  system  to  manually  inspect  and 


analyze  the  docianents  and  described  methods.  Some  assistance  can  be  derived 
from  analysis  of  execution  or  simulation  results. 


An  example  presented  in  [VANTD743  Illustrates  t »^vsls  required  for 
AC. 1(4).  Three  Implementations  of  the  equation; 

3 2 

y • Ax  + Bx  + Cx  + D 

are  provided.  They  are  listed  In  order  of  increeslng  efficiency  and  accuracy. 


ri 


y « A*X**3  + B*X**2  + C*X  + D 
y » A*X*X*X  + B*X*X  + C*X  + D 
y • D + X*(C  + X*(B+A*X)) 


i 

! 
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METRIC 

REQMTS  1 

DESIGN  1 

iw>lementation| 

SOURCE 

TOOL 

SOURCE 

TOOL 

SOURCE 

TOOL 

ET.  1 ERROR  TOLERANCE  CONTROL  CHECKLIST: 

(1)  Any  concurrent  processing  centrally 
controlled. 

PDS 

DOS 

MAN 

CODE 

NAN 

(2)  Errors  should  be  fixable  and  processing 
continued. 

PDS 

DOS 

MAN 

CODE 

MAN 

(3)  When  an  error  condition  is  detected,  it 
should  be  passed  up  to  calling  routine. 

POS 

DOS 

MAN 

CODE 

MAN 

ET.  2 RECOVERY  FROM  IMPROPER  INPUT  DATA  CHECKLIST: 

(1)  A definitive  statement  of  requirement  for 
error  tolerance  of  input  data. 

SRS 

NAN 

(2)  Range  of  values  (reasonableness)  for  items 
specified  and  checked. 

DOS 

MAN 

CODE 

MAN 

(3)  Conflicting  requests  and  illegal  conbinations 
identified  and  checked. 

DOS 

NAN 

CODE 

MAN 

(4)  All  inout  1$  checked  before  processing  begins. 

DOS 

MAN 

CODE 

HAN 

(5)  Determination  that  an  data  is  available  prior 
to  processing. 

DOS 

MAN 

CODE 

MAN 

ET.  3 RECOVERY  FROM  COMPUTATIONAL  FAILURES  CHECKLIST: 

(1)  A definitive  statement  of  requirement  for 
recovery  from  computational  failures. 

SRS 

NAN 

(2)  Loop  and  multiple  transfer  index  parameters 
range  tested  before  use. 

DOS 

NAN 

CODE 

MAN 

(3)  Subscript  checking. 

DOS 

MAN 

CODE 

NAN 

(4)  Critical  output  parameters  reasonableness 
checked  during  processing. 

DOS 

NAN 

CODE 

MAN 

ET.  4 RECOVERY  FROM  IWOHARE  FAULTS  CHECKLIST: 

(1)  A definitive  statement  of  requirement  for 
recovery  from  hardware  faults. 

SRS 

NAN 

(2)  Recovery  from  hardware  faults  (e.g., 
arithmetic  faults,  power  failure,  clock). 

DOS 

NAN 

CODE 

MAN 

ET.  5 RECOVERY  FROM  DEVICE  ERRORS  CHECKLIST: 

(1)  Definitive  statement  of  requirement  for 
recovery  from  device  errors. 

SRS 

NAN 

(2)  Recovery  from  device  errors. 

DOS 

HAN 

CODE 

NAN 

hl 

i;i 

r • I 


I 


0.6  (continued) 
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All  of  the  measures  associated  with  error  tolerance  (ET.1  through  ET.5)  were 
manually  collected.  The  major  source  of  Information  was  the  description  of 
the  Inputs,  processing,  and  limitations  In  the  detailed  design  specification. 
Often,  an  Itemized  limitation  would  contain  a phrase  such  as;  "all  function 
cards  must  be  reinput  under  an  error  condition,"  or,  "the  permissible  values 
are,"  or,  "the  acceptable  range,"  etc.  These  phrases  Identify  limitations 
on  the  processing.  In  order  for  the  system  to  be  error  tolerant.  It  should 
"gracefully"  handle  violations  of  these  limitations.  Most  of  the  measure- 
ments are  oriented  toward  the  Identification  of  failures  to  provide  typical 
error  tolerant  capabilities. 

The  software  system  requirement  specification  was  also  a valuable  source. 

In  our  environment,  an  Operational  Hardware/Software  Specification  evolves 
during  the  development  phase  from  the  System  Requirements  Specification. 

The  operational  constraints  Imposed  by  both  hardware  and  software  are  de- 
scribed. The  error  tolerant  handling  of  the  constraints  must  be  considered. 
The  manual  Inspection  of  the  code  Is  considerably  easier  If  there  1$  a good 
detail  design  document  and  accurate  design  charts.  These  documents  aid  In 
Identifying  the  key  portions  of  the  code  to  Inspect.  Basically  Input  pro- 
cessing code  should  be  examined  to  Insure  necessary  checks  occur.  Loop 
Indices  and  subscript  checking.  If  necessary.  Is  easily  checked.  The  coninon 
example  given  Is: 

IVAR  » 0 

DO  100  I - 1,N 

IVAR  • IVAR+1 


100  CONTINUE 

If  N <0,  the  wrong  value  of  IVAR  Is  realized.  Instead  a check  should  be 
made,  such  as: 


D-12 


0.6  (continued) 


IVAR  > 0 
IF  (N.LE.O) 

DO  100  I » I.N 
IVAR  = IVAR+1 


100  CONTINUE 
END  IF 
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METRIC 

Kd 

IMTS 

DESIGN 

SOURCE 

TOOL 

SOURCE 

TOOL 

SOURCE 

TOOL 

SI.  1 DESIGN  STRUCTURE  MEASURE: 

(1)  Otsign  organized  In  top  down  fashion. 

RDS 

DOS 

MAN 

(ISOS) 

CODE 

NAN 

(CA) 

(2)  No  duplicate  functions. 

(3)  Independence  of  module 

/.  f nodules  violate  ru1e\ 

^ total  1 nodules  j 

DOS 

MAN 

CODE 

MAN 

(4)  Module  processing  not  dependent  on  prior 
processing. 

/.  1 modules  violate  ru1e\ 

^ total  # nodules  j 

DOS 

MAN 

CODE 

NAN 

(S)  Each  module  description  Includes  Input,  output 
processing,  limitations. 

/.  1 modules  violate  ru1e\ 

"total  i modules — ) 

DOS 

MAN 

(POL) 

(€)  Each  module  has  single  entrance,  single  exit. 
/.  f modules  violate  ru1e\ 
y total  f modules  J 

DOS 

ISOS 

CODE 

6JS 

(7)  No  global  data. 

DOS 

MAN 

CODE 

NAN 

(CA) 

The  design  documentation  consists  of  a preliminary  design  specification  (PDS) 
and  a detailed  design  specification  (DOS)  (see  Appendix  B for  a description). 
The  detailed  design  specification  is  usually  organized  with  an  overview  and 
subsystem  descriptions  and  then  descriptions  at  the  program  level.  The 
preliminary  design  specification  and  detailed  design  overview  were  the  major 
sources  for  determining  the  design  structure  metric  (SI.l).  during  the 
design  phase.  A hierarchy  chart  and  brief  descriptions  of  the  subsystems 
and  planned  modules  were  used  to  determine  if  top  down  design  techniques  had 
been  followed.  Several  violations  were  found  for  SI.l (3),  (4),  and  (6). 

Most  of  these  violations  were  identified  by  examining  the  section  of  the 
detailed  design  specifications  which  describe  the  colling  sequence.  Several 
modules  had  multiple  entrances  (RE/ISOS  flags  this  attribute  from  the  design 
charts).  Several  modules  were  dependent  on  prior  processing.  The  most 
common  indication  of  this  was  "the  following  data  blocks  must  be  set  prior 
to  execution  of  this  module"  phrase  found  in  the  calling  sequence  description. 


WTmc 


kl.  3 COMPLEXITY  MEASURE  (by  module.  ptra.  6.2. 2. 6) 


REQNTS 


SOURCE! 


TOOL 


DESIGN 


SOURCE 


DOS 


TOOL 


ISOS 


IMPLEMENTATIOld 


SOURCE 


CUE 


TOOL 


GJS 


GE/ISOS  Mas  utilized  to  calculate  a complexity  measure  (SI. 3)  from  a special 


form  of  a design  chart.  This  measure  could  be  calculated  from  machine 
readable  design  charts*  a PIH.,  or  from  code*  however,  these  enhancements 
have  not  been  Implemented  In  ISDS  yet.  Instead*  data  gathered  by  a code 
auditing  routine*  GJSUMRY*  was  utilized  to  calculate  a modified  version  of 
Halstead's  E measure  [HALSM73,HALSM72*L0VET76a].  This  measure  Is  based  on 
the  number  of  operators  and  operands  In  a program. 

An  example  of  the  automated  calculation  of  the  ISOS  generated  complexity 
measure  is  In  Figure  D.9-1. 


t : 
[ 


Figure  D.9-1  GE/ISOS  Complexity  Measure 
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METRIC 

SI.  4 MEASURE  OF  COOINfi  SIMPLICITY  (by  mdule) 

(1)  Module  flow  top  to  bottom. 

(2)  Negotive  Boolean  or  complicated  compound 
Boolean  expressions  used. 

(♦  of  above \ 

1-  f executable  statements  ) 

(3)  Jumps  In  and  out  of  loops 

sinole  entry/sinole  exit  loops^ 

(4)  Loop  Index  modified 

(,  I loop  Indices  modified  ^ 

’ ^taT  TldSpi ) 

(5)  Module  Is  not  self -modifying. 

(6)  All  arguments  passed  to  a module  are 
parametric. 

(7)  Number  of  statement  labels. 

f,  f labels 'N 

V'  f executable  statements y 

(8)  Unique  names  for  variables. 

(9)  Single  use  of  variables. 

(10)  No  mixed  mode  expressions. 

(11)  Nesting  level 


(12)  Number  of  branches 

♦ branches 

V*  f executable  statements^ 

(13)  Number  of  GOTOs 

# GOTO  stat^nts  "S 
V*  f executable  statements y 

(14)  No  extraneous  code  exists. 

(15)  Variable  mix  in  a module 

I 4 Internal  variables 
\ total  # variables  J 


REQNTS 


DESIGN  lIMPLEMENTATION 


TOOL 

SOURCE 

TOOL 

CODE 

NAN 

CODE 

NAN 

1 

! CODE 

GJS 

CODE 

NAN 

CODE 

NAN 

CODE 

NAN 

CODE 

NAN 

(CA) 

CODE 

NAN 

CODE 

HAN 

CODE 

NAN 

CODE 

NAN 

(CA) 

CODE 

GJS 

1 

j 

CODE 

gJs 

CODE 

MAN 

CODE 

NAN 

(CA) 

CODE 

NAN 

(CA) 

(16)  Variable  density 

f,  I variables 

v'  7~executable  state 


The  code  simplicity  measurements  (SI. 4)  represent  a mix  of  manual  Inspection 
of  the  source  code  and  a report  of  statistics  about  the  source  code  by  a code 
audit  routine.  Several  other  elements  could  have  been  collected  automatically 
(as  indicated)  with  some  minor  enhancements  to  the  code  audit  routine.  Most 
of  these  manual  measurements,  once  a routine  is  established,  can  be  done  quite 
efficiently.  Approximately  1-2  man-hours  were  spent  per  routine  taking  the 
manual  measurements  relating  to  the  source  code. 


0.10  (continued) 

Figure  D.10-1  displays  examples  of  output  from  two  code  audit  routines  used. 
The  annotations  describe  some  of  the  data  collected. 

An  example  of  SI. 4(2),  complicated  BOOLEAN  expressions,  shown  here; 


IF  j^GRLERR  EQ  V(NOREC)  $ 

BEGIN  ^^BEGIN  3 

IF  /GOEREQ  NQ  V(OEL)  $ 

BEGIN  ^^BEGIN  4 ft 

IF(j‘SDACCS  EQ  V(RANFIX)  AND  /6DERRF  GR  ^SDMAXN  OR 

?«GDnDT($0$)  NQ  V(VBLK)  AND(?»GDERRF  NQ  0 OR  j^GDERRF 

NQ  0))  $ 

BEGIN  ^^BEGIN  5 ff 

Illustrates  the  complexity  It  adds  to  the  program.  Their  use  may  also  compll 
cate  the  logic  as  shown  by  [KERNB74]  with  this  example; 

IF  (.NOT.  FLAG)  THEN  A=B  ELSE  X-Y; 

which  Is  simplified  as 

IF  (FLAG)  THEN  X=Y  ELSE  A=B;. 

The  standards  and  conventions  we  established  for  JOVIAL  restrict  the  use  of 
the  switch  construct  because  It  constitutes  self-modifl cation.  An  example 
of  the  complexity  self-modifying  code  adds  Is  given  by  [Y0URE75]  with  this 
COBOL  example  (ALTER  In  COBOL  Is  similar  to  SWITCH  In  JOVIAL). 

ALTER  SWITCHl  TO  PROCEED  TO  SWITCH2 


SWITCH! 

GO  TO  INITIALIZATION-ROUTINE 


SWITCH2 

ALTER  SWITCH3  TO  PROCEED  TO  SWITCH4 
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Figure 


0.10  (continued) 

[B0EHM73a]  provides  justification  for  SI. 4(7)  with  this  example: 

The  label,  5,  is  unnecessary. 

00  10  I = l.N 
5 X(I)  = A(I)+B(I) 

10  CONTINUE 

SI. 4(8),  (9),  (10)  posed  the  biggest  data  collection  problem.  Certainly, 
examples  such  as: 

A(1)  stores  craft  altitude 

A(2)  stores  craft  velocity 

A(3)  stores  date  of  run 


TOTAL 

SUM 


all  same  quantities  used  in  different  modules 


given  in  [B0EHM73a],  and 

OIMENSION  A(20),  B(30) 

00  10  I = 1,50 
10  A(I)  = J 

by  [Y0UR75)  are  important  to  catch.  The  greatest  assistance  in  applying 
these  metrics  is  well  commented  declarative  statements  which  identify  the 
attributes  of  the  variables.  An  example  from  a module  inspected  during 
this  study  is  shown  here: 


If**********************************  iTQiis 
* ■ 


SEC 

SAVPLK 

SAVFILE 

SAVREC 

OPERANS 

SZERO 

BLEN 


0 $ ft  SAVEO  SYS  ERR  COUNT  ff 

0 8 $ ft  SAVE  BLOCK  NAME  ft 

1 48  U $ ft  SAVE  FILE  NUMBER  ft 

I 48  U $ ft  SAVE  RECORD  NUMBER  ff 
08$  ft  OPERATOR  RESPONSE  ff 
OS  ff  MUST  BE  ZERO  ff 

0 $ ft  BLOCK  LENGTH  ff 


0 6$ 
0 $ 


0.]] 


METRIC 

REC 

IKTS 

OESISM 

IlVlENENTATIONl 

SOURCE 

TOOL 

SOURCE 

TOOL 

SOURCE 

TOOL  1 

NO.  1 STABILITY  MEASURE 

/ «xD€cted  * modules  changed  ) 

V total  ^ moOulOs  / 

POS 

DOS 

NAN 

(ISOS) 

■ 

■ 

The  stability  measure  Is  based  on  Myer's  [MYER675]  stability  model.  Each 
module  Is  categorized  according  to  specific  criteria  as  to  Its  module 
strength  and  module  coupling.  A matrix  Is  built  based  on  values  assigned 
the  various  categorizations  and  the  module  Interactions  within  the  system. 
Taking  Into  account  the  various  paths  by  which  a change  to  one  module  may 
effect  another  module,  a second  order  matrix  Is  calculated.  From  this  second 
order  matrix,  a prediction  of  the  number  of  modules  that  can  be  expected  to  be 
effected  If  any  module  Is  changed  Is  calculated.  Since  this  Is  a system 
level  metric,  yalldatiQn  was  Impossible.  The  value  derived  for  System  B, 
for  example  was  very  high  compared  to  the  examples  presented  by  Myers.  This 
Is  very  logical  because  In  our  environment,  most  data  Is  global  to  the 
system  (COMPOOL)  and  accessed  by  many  routines.  Thus  any  change  to  the 
data  or  the  way  the  data  Is  manipulated  by  a routine  potentially  can  effect 
a large  number  of  other  modules  In  the  system.  The  significance  of  the 
measure  Is  even  If  other  modules  are  not  effected,  considerably  more  effort 
must  be  expended  testing  to  Insure  they  are  not  In  this  type  of  an  environment. 
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SOURCE 
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SOURCE 

TOOL 

POS 

DOS 
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(ISOS) 

CODE 

NAN 

(CA) 

CODE 

GJS 

DOS 

NAN 

CODE 

NAN 

DOS 

MAN 

CODE 

MAN 

DOS 

NAN 

CODE 

NAN 

DOS 

NAN 

CODE 

MAN 

DOS 

MAN 

CODE 

MAN 

DOS 
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CODE 

NAN 

(CA) 

MO.  2 MOOUUR  IMPLEMENTATION  MEASURE 

(1)  Hierarchical  structure 
/.  # violations  of  hierarchy ^ 

V ‘ total  # modules  / 

(2)  All  nodules  do  not  exceed  standard  nodule 
size  (100) 


(' 


I nodules > 100) ^ 


total  If  nodules 

(3)  All  modules  represent  one  function 

(1  * modules  violate  rule\ 
tota)  I modules  / 

(4)  Controlling  parameters  defined  by  calling 
nodule 

I modules  violate  ruleN 
total  # modules  J 

(S)  Input  data  controlled  by  calling  module 
f.  I modules  violate  ru1e'\ 

V " total  I modules  J 

(6)  Output  data  provided  to  calling  module 

(,  0 modules  violate  >-uIe\ 
total  f modules  / 

(7)  Control  returned  to  calling  nodule 
f modules  violate  rule') 

V * total  0 modules  J 

(8)  Modules  do  not  share  temporary  storage 


Each  of  the  modular  implementation  measures  except  for  MO. 2(2),  were  collected 
manually  during  this  effort.  The  detail  design  specifications  identified 
module  interactions  by  a cal led/ cal ling  matrix  and  had  a specific  section 
which  described  the  calling  sequence  and  parameters  for  each  module.  An 
evaluation  of  the  parameters  and  their  effect  on  the  processing  led  to  the 
above  measures. 


? 


I 

i 

f 

i 


In  MO. 2(1),  any  interactions  between  modules  which  are  not  successors  or 
predecessors  on  the  same  branch  are  considered  violations.  In  Figure  D.12-1 
the  measure  is  3 violations/8  modules  * .375.  The  violations  are  the  inter- 
action between  modules  1-2,  between  1-5  and  2-5,  and  between  5-6. 


D.13 


This  measurement  was  also  easily  determined  by  the  same  source  mentioned 
In  D.12  and  confirmation  made  by  Inspection  of  the  code. 

The  Implication  of  this  measure  Is  that  modules  which  interface  with  a number 
of  other  modules  1n  their  current  application  will  probably  be  easier  to 
Interface  or  use  In  another  application. 
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6E.  2 IHPLEMENTATION  FOR  GENERALITY  CHECtCLIST 

(1)  Input,  processing,  output  functions  are  not 
Mlied  In  a single  module. 

I modules  violate  rule'l 
' total  H modules  J 

(2)  Application  and  Mchlne-depcndent  functions 
are  not  mixed  In  a single  module. 

C- 

(3)  Processing  not  data  volume  limited. 

/,  # modules  limited  \ 

V"  total  f nodules  J 


DOS 


MAN 

(ISOS 


(4)  Processing  not  data  value  limited. 

(,  I modules  limited  \ 
total  # modules  J 

(5)  All  constants  should  be  defined  once. 

(,  I modules  violate  rule  A 
total  # moduUs  ) 


DOS 
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CODE 


CODE 


CODE 


CODE 


NAN 

(CA) 


NAN 

(CA) 


MAN 

(EA) 

NAN 

(EA) 

MAN 

(CA) 


Since  GE.2  is  a system  level  metric,  no  validation  was  possible.  However, 
the  characteristics  identified  by  these  measures,  we  feel,  are  quite 
important  to  the  generality  of  the  software  produced.  (1)  and  (2)  were 
determined  by  a simple  identification  in  the  design  documents  and  code 
whether  the  module  contained  any  machine  dependent,  input,  and/or  output 
code.  The  more  modules  involved  in  machine  dependent  functions  or  input 
or  output,  the  more  effort  will  be  required  to  use  any  one  module  in 
another  environment  or  application. 


GE.2  (3)  and  (4)  correspond  to  several  error  tolerant  elements  where  the  intent 
of  the  measures  was  to  determine  how  well  the  system  handles  exception  cases. 
These  measures  are  related  to  the  fact  that  there  are  exception  cases.  The 
more  limitations  on  a module  or  a system  the  less  generally  usable  it  is.  This 
includes  the  case  where  the  algorithm  or  function  and  its  implementation  are 
restrictive  or  the  case  where  poor  programming  practices  have  restricted 
the  general  use  of  the  program.  An  example  of  the  later  situation  is  if 
the  limit  of  a loop  in  a module  which  represents  the  maximum  number  of  inputs 
to  that  module  is  "hard- coded",  it  is  more  difficult  to  change  the  module 
correctly  to  handle  more  input. 


ri 


D.16 


METRIC 

REC 

|MTS 

KSIGN  1 

lliPLElKNTATIONl 

SOURCE 

TOOL 

SOURCE 

TOOL 

SOURCE 

TOOL 

EX.  1 DATA  STORAGE  EXPANSION  MEASURE: 

(1)  Losical  processing  Independent  of  storage 
specIffcatton/requfrWnents  (by  aMdiile) 

ft  1 awdules  violate  rule  T 

U total  f nodules  ) 

(2)  Percent  of  memory  capacity  uncoMiltted 

f aiwunt  of  memory  uncoanltted  \ 

^ total  ampunt  of  available  memory  J 

ods 

MAN 

CODE 

coot 



MAN 

(EA) 

EX. 1(2)  requires  execution  of  the  code  undet*  typical  loading  conditions. 
Most  Job  execution  reports  provided  by  operating  system  software  provides 
the  data  to  determine  this  measure. 


I Examples,  from  System  B and  [Y01JRE75],  of  using  parameters  to  Insure  the 

[ processing  Is  Independent  of  the  storage  specification  follow: 

( 

FOR  N = NENT(TRELOC)  $ 

BEGIN 


END 

*THIS  ALC  PROGRAM  REQUIRES  THREE  TABLES:  THE  SIZE  OF 
♦EACH  TABLE  IS  A FUNCTION  OF  THE  PARAMETER 
*"SIZE".T0  CHANGE  THE  SIZE  OF  THE  TABLES, 

*MERELY  REDEFINE  “SIZE". 

★ 

' 1 SIZE  EQU  40 

;]  TABLE!  bS  CL(StZE) 

‘ j TABLE2  OS  CL(2*S1ZE) 

TABLES  DS  CL(SIZEfB) 

i 

S 
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EX.  2 EXTENSIBILITY  MEASURE: 

(1)  Accuracy,  convergence,  timing  attributes 
which  control  processing  are  parametric. 

DOS 

MAN 

CODE 

MAN 

# moiiules  violate  rule  T 

V total  ' modules  J 

(2)  Modules  table  driven. 

DOS 

MAN 

CODE 

MAN 

ft  t modules  not  table  driven ^ 
total  A modules  J 

(3)  Percent  of  speed  capacity  uncommitted. 

CODE 

(EA) 

f amount  of  cycle  time  unconnltted 

V total  processing  time  / 

The  discussion  In  paragraph  D.15  pertains  to  EX. 2(3)  also.  EX. 2(1)  and  (2) 
require  an  analysis  of  the  design  strategy  expressed  In  the  design  document. 
The  following  example  Is  described  In  [Y0URE75]. 

c 


1 


c 

c 

c 

c 

c 

c 

c 

c 

c 


THIS  IS  A SUBROUTINE  THAT  WILL  SEARCH  THROUGH  ANY 
SINGLE-DIMENSIONED  ARRAY  TO  FIND  A SPECIFIED 
ARGUEMENT.  THE  ARGUEMENTS  TO  THE  SUBROUTINE 
ARE  AS  FOLLOWS; 

TABLE  THE  NAME  OF  THE  ARRAY  TO  BE  SEARCHED 

FIRST  THE  LOWER  DIMENSION  OF  THE  ARRAY 

LAST  THE  UPPER  DIMENSION  OF  THE  ARRAY 

ARG  THE  QUANTITY  BEING  SEARCHED  FOR 

FLAG  INDICATES  WHETHER  OR  NOT  SEARCH  WAS  SUCCESSFUL 


SUBROUTINE  SEARCH  (TABLE,  FIRST,  LAST,  ARG,  FLAG) 
DIMENSION  TABLE  (FIRST,  LAST) 
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IN.  1 MOOUIC  TESTING  MEASURE  (by  modulo) 

(1)  Path  covcroye.  v 

poths  to  bo  tested  \ 
total  I paths  / 

(t)  All  Input  parameters  boundary  tested. 

/I  parameters  to  be  boundary  tested \ 

^ total  # parameters  J 

IN.  2 INTEGRATION  TESTING  MEASURE 

(1)  Module  Interfaces  tested. 

(I  to  be  tested  v 
total  Interfaces) 

(2)  Performance  requireamnts  (timing  t storage) 
coverage. 

(I  requirements  to  to  testedx 
total  I perT  requirements  I 

IN.  3 SYSTEM  TESTING  MEASURE 

(I)  Nodule  coverage  (for  all  test  scenarios) 

/ 1 modules  to  be  tested  \ 

I total  I of  modules  f 

(2)  Identification  of  cost  Inputs  and  outputs  In 
sunmry  form. 
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MAN 


CODE 

CODE 

CODE 

CODE 

CODE 

CODE 


(AVS) 

(TPL) 

(AVS) 

(TPL) 


(AVS) 

(TPL) 


(AVS) 

(TPL) 

MAN 


For  both  system  developments  used  In  this  study,  module  testing  was  docu 
mented  In  Individual  progranmer' s notebooks.  The  notebooks  contain  the 
following  Information: 


e Date  of  test 

a Brief  staternent  of  test  objectives 
e Brief  description  of  Input  conditions 
a Description  of  test  results  and  analysis 

The  notebooks  were  not  available  for  the  tv«o  system  developments  because  of 
the  timeframe  since  delivery  of  the  two  systams.  A review  of  notebooks  of 
current  software  developments  revealed  that  sufficient  Information  Is 
available  to  determine  the  IN.1  measures.  Several  automated  aids  are  available 
In  this  area  and  are  discussed  In  paragraph  8.3.  An  example  of  automated 
assistance  during  the  design  phase  Is  the  minimum  number  of  test  cases  to 
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cover  all  program  paths  determined  by  GE/ISDS  from  design  charts  of  a 
program.  An  example  report  is  shown  In  Figure  D.17-1. 
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4 Doos  (ho  lost  of  ino  date  for 
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4 Doos  tPM  lost  of  tn#  doto  for  t. 
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Figure  D.17-1  Minimum  Test  Case  Generation  by  GE/ISDS 
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i The  Interface  control  document  was  a primary  source  for  Identifying  a11  of 

I the  data  files  used  as  Interfaces.  They  are  described  In  detail  In  this 

' document.  The  validation  and  acceptance  test  specification  Indicated  that 

testing  module  Interfaces  (IN. 2(1))  Is  a primary  objective  of  development 
testing.  Included  with  this  specification  Is  the  development  test  plan 
which  covers: 

e Statement  of  function  tested 
e Modules  exercised 
e Data  base  value  required 
e Inputs 

e Expected  output 
e Analysis  of  output 
e Priority  of  test 

> The  performance  requirements  (IN. 2(2))  are  extracted  from  the  software 

requirements  specification.  They  are  traced  by  a routine  to  Insure 
I compliance.  An  example  output  of  the  routine  Illustrates  several  performance 

requirements  Identified  relating  to  a particular  specification  follows: 


1 


SPECIFICATION  TO  PERFORMANCE  REQUIREMENT  TRANSLATION 

SPECIFICATION  The  executive  function  shall  provide  the  option  for 

specific  parameter  status  reporting  and  data  base  update. 
PERFORMANCE  EX-40 

REQUIREMENTS  executive  will  provide  an  operator  Interaction 

option  to  display  the  contents  of  specific  para- 
meters Identified  in  the  data  base  defined  status 
table. 

EX-41 

The  executive  will  provide  an  operator  Interaction 
option  to  data  base  update  specific  parameters 
identified  In  data  base  defined  status  table. 

EX-42 

The  executive  will  provide  an  option  to  display  para- 
meter status  values  on  the  console  or  printer. 

EX-43 

The  executive  will  accept  and  Interpret  run-ccntrol 
data  Inputs  which  define  specific  parameters  to  be 
displayed  automatically,  and  the  Interval  at  which 
they  are  to  be  displayed. 

EX-44 

The  executive  will  provide  an  option  to  automatically 
display  specific  parameters  available  for  status 
reporting. 

At  the  system  level  (IN. 3),  the  validation  and  acceptance  test  plan  pro- 
vides a test  requirements  satisfaction  matrix  from  which  IN. 3(1)  can 
be  determined. 
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so.  1 QUANTITY  Of  COMMENTS  (by  module) 

/#  of  comments  (nonblank]  \ 

1 total  I lines  (nonblank) J 

SO.  2 EFFECTIVENESS  OF  COMMENTS  MEASURE 

(1)  Nodules  have  standard  formated  prologue 
comnents  which  describe: 

• Module  name/verslon  number 

• Either 

- Date 

- Purpose 

- Inputs 

- Outputs 

- Function 

- Assumptions 

- Limitations  and  restrictions 

- Accuracy  requirements 

- Error  recovery  procedures 

- References 

I modules  violate  rule  \ 
total  f modules  / 

(2)  Coanents  set  off  from  code  In  uniform  manner. 

1,  1 modules  violate  rule  \ 
v'  total  1 modules  / 

(3)  All  transfers  :f  control  A destinations 
comnented. 

f modules  violate  ru1e\ 

V total  # modules  / 

(4)  All  machine  dependent  code  commented. 

I^  i modules  violate  rule! 

\ total  # modules  / 

(5)  All  non-standard  HOL  statements  commented. 

/i  f modules  violate  rule! 

V total  f modules  / 

(6)  Attributes  of  all  declared  variables  comnented. 

It  f modules  violate  rule! 

V total  1 modules  / 

(7)  Comments  do  not  just  repeat  operation 

1,  f modules  violate  ruleX 

V ‘otal  1 modules  / 
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(CA) 

S0.1  was  simply  derived  from  our  code  audit  routine*  a sample  output  of 
which  Is  shown  In  Figure  DJ8*1. 

SD>2  itmasures  were  all  determined  manually  from  the  source  code  listings  but 
could  be  collected  by  an  enhanced  code  audit  routine.  Some  selected  examples 
from  the  source  code  relating  to  SD.2  measurements  arc  shown  In  Figure  D.18-2. 
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Comment  Count  By  Code  Audit  Routine 


ATTRtWITES  OF  ALL  OEUAKO  VAAIASLCS  COMCNTEO 


Effectiveness  of  Comnents  Examples 


METRIC 


SO.  3 DESCRIPTIVENESS  OF  IMPLEMENTATION  LANGUAGE 
MEASURE 

(1)  High  order  language  used. 

/■,  I modules  with  direct  code  \ 
l ‘ total  t modules  / 


REQMTS  DESIGN 

source!  TOOL  source!  tool 


/,  fjnodul 
!'■  tot 

(2)  Standard  format  for  organization  of  modules. 

/,  I modules  violate  rule! 

\ " total  * modures  / 

(3)  Variable  names  (mnemonic)  descriptive  of 
physical  oh  functional  property  represented. 

I modules  violate  rule  I 
! total  # nodules  ) 

(4)  Source  code  logically  blocked  and  Indented. 

/,  > modules  violate  rule\ 
r total  # modules  / 

(5)  One  statement  per  line. 

# continuations  + multiple 

(,  statement  lines \ 

\'  total  I lines  ) 

(6)  No  language  keywords  used  as  names. 

/,  I modules  violate  rule  \ 

^ * total  I modules  / 


SD.3(1)  and  (5)  were  collected  by  the  code  audit  routine.  The  remainder  were 
determined  manually.  An  example  of  compliance  with  SD.3(3)  follows: 

ITEM  SAVBLK  » S t SAVE  BLOCK  NAME 

ITEM  SAVFILE  I 48  U $ j*?*  SAVE  FILE  NUMBER 

ITEM  SAVREC  I 48  U $ /?<  SAVE  RECORD  NUMBER 

ITEM  OPERANS  8 8$?*/  OPERATOR  RESPONSE  /?* 


IMPLEMENTATION 

SOURCE 

TOOL 

CODE 

1 

GJS 

CODE 

HAN 

(CA) 

CODE 

HAN 

CODE 

NAN 

CODE 

GJS 

CODE 

MAN 

(CA) 

which  are  obviously  better  than  variable  names  such  as 


, A0(I2,  A003,  etc. 


But  more  subtle  problems  can  arise  from  poor  variable  names  as  pointed  out 
in  this  example  in  [KERNB74]. 


WRITE  (6,60)  N.NN.NNN 


p.lf  (cenMmiad) 

In  tMs  exiMple  the  typogr«phlci|  errpf  in  tM  tin#  WWlfl  h#  to 

M|te  erxi  difficult  to  flful  because  of  the  eeMing.  A sugghtteA  WMilng  scheM 
Is  N,  HSQ,  HCUBE. 


Itest  structured  progranmlng  preproeessort  prpvl^  the  indiPtitlon  enphaslzed 
in  sp.3(4)t  Also,  the  paragraphing  cooitructs  luph  is  PO  »»»  CONTHUE  in 
FORTRAN.  PEPIN  ...  ENP  In  ALGOL  and  JOVIAL,  tht  PPt  ...  PNOs,  8E6IN;  ...  ENP; 
ahd  PROCEpURE;  ...  ENP;  in  PL/1  should  be  toMh  adVINtage  of  to  proisote  self- 
descriptiyeness.  Whether  a module  Is  ImplMiefited  with  these  techniques  in 
mind  or  not,  is  easily  discernible  by  inspectiph  Pf  the  source  code. 
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METRIC 

REQMTS 

DESIGN 

implementation! 

SOURCE 

TOOL 

SOURCE 

TOOL 

SOURCE 

TOOL 

EE.  1 PERFORMANCE  REQUIREMENTS  ALLOCATED  TO  DESIGN 

PCS 

RTR 

DOS 

(PSL) 

(HX.) 

EE.  2 ITERATIVE  PROCESSING  EFFICIENCY  MEASURE: 

(by  module) 

(1)  Non-loop  dependent  computations  kept  out  of 

DOS 

MAN 

CODE 

MAN 

loop. 

(CA) 

/.  i nonloop  dependent  statements  in  loop  \ 

total  A loop  statements  ' 

(2)  Performance  optimizing  compiler/assembly 

CODE 

MAN 

language  used. 

GdS 

(3)  Compound  expressions  defined 

CODE 

MAN 

1 compound  expression  derined  more 

(CA) 

than  once  \ 

V 1 compound  expressions  / 

(4)  Number  of  overlays. 

DOS 

MAN 

CODE 

MAN 

/ . ’ \ 

(EA) 

V 1 of  overlays  • 

(5)  Free  of  bit/byte  packing/unpacking  in  loops. 

DOS 

MAN 

CODE 

MAN 

(CA) 

(6)  Free  of  nonfunctional  executable  code. 

CODE 

MAN 

/.  A nonfunctional  executable  codev 

( total  executable  statements  / 

(7)  Decision  statements  efficiently  coded. 

CODE 

NAN 

/,  A inefficient  decision  statements  \ 

(EA) 

V total  A decision  statements  f 

(8)  Module  linkages. 

CODE 

(EA) 

/.  module  linkage  time  \ 

\'  execution  time  / 

(9)  OS  linkages. 

CODE 

(EA) 

/,  OS  linkage  time  \ 

\ execution  time  / 

EE.  3 DATA  USAGE  EFFICIENCY  MEASURE;  (by  module) 

(1)  Data  grouped  for  efficient  processing. 

DDS 

MAN 

CODE 

MAN 

DBMP 

(DBO) 

(2)  Variables  initialized  when  declared. 

CODE 

NAN 

1 A initialized  when  declared) 

(CA) 

\ total  A variables  i 

(3)  No  mix-mode  expressions. 

CODE 

MAN 

/ A mix  mode  expressions  \ 

(CA) 

i executable  statements/ 

(4)  Connon  choice  of  units/type. 

DDS 

NAN 

CODE 

MAN 

1 (1/#  occurrences  of  uncoRinon  unit  operations) 

(CA) 

(5)  Data  indexed  or  referenced  for  efficient 

DOS 

HAN 

CODE 

MAN 

processing. 

OGNP 

(080) 

(CA) 

D.20  (continued) 


Paragraph  0.17  Illustrated  how  the  performance  requlremehts  are  kept  track 
of  and  paragraph  D>1  described  the  SRS  requirements  satisfaction  vs  routine 
matrix  In  the  preliminary  design  specification.  These  sources  Identify 
the  performance  requirements,  and  the  detailed  design  specifications  reveal 
whether  they  are  accomnodated  (EE.1).  for  example,  estimates  of  storage 
requirements  and  run  times  are  documented  In  the  design  documents.  These 
estimates  which  provide  a best  estimate,  as  well  as  a hlght  and  a low 
versus  the  maximum  allowable  are  based  on  specific  scenarios.  The  SRS  might 
specify  a certain  transaction  type  must  be  handled  within  a certain  time 
frame.  The  run  time  estimates  for  the  series  of  modules  which  process 
that  transaction  type  should  comply  with  the  requirement. 

Six  of  the  measures  for  EE. 2 and  EE. 3 can  be  taken  from  the  detailed  design 
documents.  All  of  the  measures  can  be  determined  from  the  source  code.  Code 
audit,  execution  analysis,  and  data  base  analysis  all  contribute  to 
measurements  for  these  metrics.  The  most  common  violations  to  the  efficiency' 
oriented  characteristics  that  these  metrics  represent  were  related  to  EE. 2(1) 
and  (3).  Some  examples  are: 

FOR  I = l.N 
BEGIN 

AREA(I)  - 2*PI*(I+X)**2 
END 

The  2*PI  Is  an  unnecessary  computation  within  a loop. 

As  Illustrated  In  [VANTD74]  the  following: 

SIGMAl  - SIN(THETA)  + SIN(THETA)**Z 
SI6MA2  » SIN(THETA)/3.0 
Is  much  more  efficient  If  Implemented  as: 

RHO  - SIN (THETA) 

SIGMAl  - RH0+RH0**2 
SIGMA2  - RHO/3.0 
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NTTRIC 

REQNTS 

DESIGN 

implementation! 

SOURCE 

TOOL 

SOURCE 

TOOL 

SOURCE 

TOOL 

SE.  1 STORAGE  EFFICIENCY  MEASURE;  (by  nodule) 

(1)  Storage  requirements  allocated  to  design. 

DOS 

MAN 

(PSL) 

(POL) 

(2)  Virtual  storage  facilities  used. 

DOS 

NAN 

CODE 

MAN 

(3)  Cannon  data  defined  only  once. 

A # variables  defined  more  than  once  \ 

V total  f variables  / 

CODE 

MAN 

(4)  Program  segmentation. 

/.  maximum  segment  length  \ 

\'  total  program  length  / 

DOS 

NAN 

CODE 

(EA) 

(S)  Data  segmentation. 

amount  of  unused  data  \ 

\'  total  amount  of  data  } 

DOS 

NAN 

CODE 

(EA) 

(6)  Dynamic  memory  management  utilized. 

DOS 

NAN 

CODE 

NAN 

(7)  Data  packing  used. 

COOE 

MAN 

(CA) 

(8)  Free  of  nonfunctional  code. 

1,  t nonfunctional  statements  \ 

V total  1 statements  / 

CODE 

MAN 

(9)  No  duplicate  codes. 

1 duel  1 cate  statements  \ 

\ total  H statements  / 

CODE 

NAN 

COOE 

NAN 

(10)  Storage  optimizing  compller/assembly  language 
used. 

COOE 

MAN 

6JS 

(11)  Free  of  redundant  data  elements. 

A # redundant  data  elements  \ 

V # data  elements  / 

COOE 

MAN 

I 


The  programming  environment  for  Systems  A and  B did  not  provide  capabilities 
such  as  virtual  storage,  optimizing  compilers,  or  dynamic  memory  management. 
However,  a very  restrictive  amount  of  memory  available  encouraged  many 
efficiency  techniques  to  be  utilized.  These  measures  attempt  to  identify 
poor  practices. 
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0.22 

OES 

16N 

INPLENCaTATIONl 

MITIIC 

SOWtCE 

TOOL 

TOOL 

SOURCE 

TOOL 

AC.  1 ACCESS  CONTROL  CHECKLIST: 

tl)  \lier  I/O  access  controls  provided 

SRS 

NAN 

PDS 

NAM 

coot 

MAN 

(10' s,  passwords). 

DOS 

(E)  Oatfe  base  access  control  provided 

SNS 

NAN 

PDS 

NAN 

CODE 

NAN 

(authorization  tables,  privacy  locks). 

SOS 

(3)  Hemory  protection  across  tasks  provided. 

SRS 

NAN 

PDS 

NAN 

CODE 

MAN 

DOS 

AA.  1 ACCESS  AUDIT  CHECKLIST: 

(1)  Provisions  for  racording  and  reportldg  accesk. 

SRS 

NAh 

PDS 

NAN 

CODE 

HAN 

DOS 

(2)  Provisions  for  lamedlate  Indication  of  octets 

SRS 

NAN 

PDS 

NAN 

CODE 

MAN 

violation. 

DOS 

Neither  SystM  A hor  B had  any  specified  requirement  fbr  access  contrbi 
capabilities.  Therdfore,  each  bf  these  measures  mere  nbt  applicable. 
The  Identified  sources  would  nbhnally  contain  statemehts  which  would 
provide  the  required  data. 
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REQNTS 

DESIGN 

implementation! 

SOURCE 

TOOL 

SOURCE 

TOOL 

SOURCE 

TOOL 

OP.  1 OPERABILITY  CHECKLIST: 

(1)  All  steps  of  operation  described 
(normal  and  alternative  flows). 

SRS 

MAN 

UOH 

MAN 

UOM 

MAN 

(Z)  All  error  conditions  and  responses 
appropriately  described  to  operator. 

SRS 

NAN 

UOM 

NAN 

UOM 

HAN 

(3)  Provisions  for  operator  to  Interrupt,  obtain 
status,  save,  modify,  and  continue  processing. 

SRS 

HAN 

UQN 

MAN 

UOM 

MAN 

(4)  NuR^r  of  operator  actions  reasonable. 

/.  time  for  operator  actions! 

\ total  time  for  job  } 

CODE 

(EA) 

(S)  Job  set  up  and  tear  down  procedures  described. 

UOM 

HAN 

(6)  Hard  copy  log  of  Interactions  maintained. 

UOM 

MAN 

UOM 

MAN 

(7)  Operator  messages  consistent  and  responses 
standard. 

UOM 

MAN 

UOM 

MAN 

The  user's  manual  or  operator  manual  (which  we  call  the  Computer  Programs 
Operating  Instructions)  Is  the  primary  source  for  these  measures.  Generally 
the  user's  manual  evolves  during  the  development  phase,  becoming  more 
detailed  and  precise  as  more  detailed  Information  on  the  exact  operating 
procedures  becomes  available.  A separate  section  contains  the  dally  usage 
Instructions  and  another  section  Identifies  and  explains  each  error  message. 

A specific  example,  paraphrased  from  the  SRS  of  System  B Is  the  following 
Itemized  requirement: 


1 


* N 


) 


The  system  should  recognize  error  conditions  and  automatically 
abort  In  an  unrecoverable  situation,  or  In  a situation  where 
recovery  Is  possible,  allow  operator  Intervention  to  either: 

e abort  the  program 

• accept  error  and  continue  the  program 

• correct  error  and  continue  the  program. 
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METRIC 

DESIGN 

inrlencntatiom 

mm 

TOOL 

22^ 

TOOL 

source 

TOOL 

TR.  1 TRAINING  CHECKLIST; 

(1)  lesson  pUn$/tra1n1ng  materiel  developed  for 
operators,  end  users,  maintalners. 

(2)  Realistic  simulated  exercises  provided. 

(3)  Sufficient  'help'  and  diagnostic  infornatlon 
available  on-line. 

1 

1 

DOS 

1 

i 

NAN 

HAN 

NAN 

(EA) 

Normally  a training  manual,  course  material,  and  lesson  plans  would  be  ; 

available  to  evaluate  and  provide  a measures  for  this  metric.  In  the  1 

i 

environment  of  the  two  systems  of  this  study,  personnel  quite  familiar 
with  the  systems  operated  them.  Thus  much  less  formal  documentation  j 

was  available.  The  user's  manual  had  a section  on  training  and  Intro-  \ 

ductory  Information  on  the  systems.  These  measures  are  manually 
extracted  from  the  Identified  sources. 

An  excellent  example  of  compliance  with  the  TR.1(3)  measure  Is  found  In 
the  support  software,  Gt/ISDS.  At  any  level  of  the  system,  ’HELP’  can 
[ be  typed,  and  a list  of  correct  commands  at  that  level  Is  provided  the 

I on-line  user. 
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REQMTS 

0ESI6N 

implementation] 

SOURCE 

TOOL 

SOURCE 

TOOL 

SOURCE 

TOOL 

CM.  I USER  INPUT  INTERFACE  MEASURE: 

PI 

(1)  Default  values  defined. 

( # defaults  \ 

(total  # parameters  / 

DOS 

UON 

MAN 

(2)  Input  formats  uniform. 

( 1 \ 

DOS 

MAN 

UON 

MAN 

V f different  Input  record  formats  / 

(3)  Each  Input  record  self  identifying. 

/.  # that  are  not  self  Identifyino) 

'total  # input  records  / 

DOS 

MAN 

UOM 

MAN 

(4)  Input  can  be  verified  by  user  prior  to 
execution. 

DDS 

NAN 

UOM 

NAN 

(5)  Input  terminated  by  explicitly  defined 
logical  end  of  Input. 

DOS 

NAN 

UOM 

HAN 

(6)  Provision  for  specifying  Input  from 
different  media. 

SRS 

MAN 

DDS 

HAN 

UOM 

MAN 

CM.  2 USER  OUTPUT  INTERFACE  MEASURE: 

(l)  Selective  output  controls. 

SRS 

MAN 

DDS 

MAN 

UOM 

MAN 

(2)  Outputs  have  unique  descriptive  user 
oriented  labels. 

DDS 

MAN 

UOM 

MAN 

(3)  Outputs  have  user  oriented  units. 

DDS 

NAN 

UOM 

MAN 

(4)  Uniform  Output  formats. 

1 1 \ 

DOS 

MAN 

UOM 

MAN 

(S)  Logical  groups  of  output  separated  for  user 
examination. 

DOS 

MAN 

UOM 

NAN 

(6)  Relationship  between  error  messages  and 
outputs  Is  unambiguous. 

DOS 

NAN 

UOM 

HAN 

(7)  Provision  for  reducing  output  to  different 
media. 

DDS 

MAN 

UOM 

MAN 

Manual  reviews  of  the  detailed  design  specifications  and  users  manual 
provided  the  data  for  all  of  these  measures.  Each  Input/output  format  was 
described  In  the  users  manual.  The  SRS  Identified  the  requirements  for 
different  modes  of  operation  (which  Involved  different  I/O),  from  cards, 
taipe,  and  teletype,  and  for  output  compression  and  expansion. 
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0.25  (continued) 


The  selective  output  controls  should  Include  selective  debug  options  In 
case  the  program  exhibits  unusual  behavior.  The  output  labels  and  units 
are  almost  as  valuable  to  the  user  as  the  results  theaiselves.  Error  messages 
should  be  helpful,  for  example: 

PARAMETER  *******  MUST  BE  BCI 

LOGICAL  UNIT  ***  IS  NOT  VALID 

DATA  BASE  ****  WAS  NOT  FOUND  IN  THE  DISC  DIRECTORY 

VERB  ******  IS  NOT  LEGAL  FOR  FUNCTION. 
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METRIC 

REQMTS 

DESIGN 

implementation! 

SOURCE 

TOOL 

SOURCE 

TOOL 

SOURCE 

TOOL 

SS.  1 SOFTWARE  SYSTEM  INOEPENOEIKE  MEASURE: 

i 

(1)  Dependence  on  software  system  utility  programs. 
/•.  # programs  • utility  prooramV 
\'  total  » programs  f 

DOS 

MAN 

CODE 

MAN 

(CA) 

(?)  Dependence  on  software  system  library  routines. 
A # library  routines  used  \ 

V total  # nodules  / 

DOS 

MAN 

CODE 

MAN 

(CA) 

(3)  Common,  standard  subset  of  languaoe  used. 

A # module  violate  rulel 

V total  » modules  / 

DOS 

MAN 

CODE 

MAN 

(CA) 

(4)  Free  from  operating  system  references. 

A 1 modules  with  OS  references) 

V total  1 modules  / 

DOS 

MAN 

CODE 

MAN 

(CA) 

These  measures  were  taken  manually  from  the  code  and  design  documents. 
Evaluation  of  the  execution  or  c mpilation  of  a routine  also  is  helpful. 
In  a particular  environment,  automated  identification  of  these  measures 
would  be  possible. 


METRIC 


REQMTS 


SOURCE  TOOL 


DESIGN 


SOURCE  TOOL 


IMPLtMENTATIOW 


SOURCE  TOOL 


MI.  1 MACHINE  INOEPENOENCE  MEASURE: 

(1)  Programming  language  used  available  on  other 
machines. 

(2)  Free  from  Input/output  references. 

I modules  with  I/O  references  \ 


DOS 

ODS 


MAN 

MAN 


CODE 

CODE 


1- 


total  V modules 


MAN 


MAN 

(CA) 


(3)  Code  1$  Independent  of  word  and  character  size 

/i  f modules  violate  ru1e\ 

V total  9 modules  7 

(4)  Data  representation  machine  Independent. 

/i  f modules  violate  ru1e\ 

\ total  * modules  / 


CODE  MAN 

CODE  MAN 


j 


Specific  JOVIAL  constructs  such  as  BIT/BYTE,  the  I/O  system  routines, 
and  DIRECT  code  were  keyed  on  while  searching  the  code  to  determine  these 
measures. 
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METRIC 


C.  1 COMMUNICATIONS  COMMONALITY  CHECKLIST: 

(1)  Definitive  stateinent  of  requirement  for 
conrr-jnicatlon  with  other  systems. 

(2)  Protocol  standards  established  and  followed. 


(3)  Single  module  interface  for  Input. 
1 


(4)  Single  module  Interface  for  output. 
1 


modules  used  for  output 


REQMTS  0ESI6N  IMPLEMENTATION 


SOURCE  TOOL  SOURCE  TOOL  SOURCE  TOOL 


SRS  MAN 


CODE  MAN 


DOS  MAN  CODE  MAN 

(PDL)  (CA) 

DOS  MAN  CODE  MAN 

(PDL)  (CA) 


The  interface  control  document  specifically  identifies  and  describes  any 
interfaces  between  systems.  This  is  especially  important  in  an  associate 
contractor  environment.  The  detail  design  specification  and  the  code 
were  utilized  to  identify  the  routines  which  contained  any  I/O  operations 
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DESIGN 

implementation! 

SOURCE 

TOOL 

SOURCE 

TOOL 

source 

tool 

CO.  1 HALSTEAD'S  MEASURE  (by  module) 

[I'lmodule  lenotli  calculated-module  lenqth  observed  \ 

code 

GJS 

\ 1 module  length  observed  / 

The  number  of  operators  and  operands  were  collected  by  the  code  audit 
routine.  A routine  was  written  to  calculate  the  metric  based  on  those 
Inputs. 


'i 


D-S1/D-S2 


MISSION 

of 

Rome  Air  Development  Center 


RADC  plans  and  conducts  research,  exploratory  and  advanced 
developaaent  programs  in  ctxmand,  control,  and  comiunlcatlons 
(C^)  activities,  and  in  the  areas  of  information  sciences 
and  intelligence.  The  principal  technical  mission  areas 
are  communications,  electromagpnetic  guidance  and  control, 
surveillance  of  ground  and  aerospace  objects,  intelligence 
data  collection  etnd  handling,  information  system  t'%chnology, 
ionospheric  propagation,  solid  state  sciences,  microwave 
physics  anc  electronic  reliability,  maintainability  and 
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